Transcript of the talk, from the video's captions. Auto-generated: speaker names in particular are unreliable. ======================================================================== # Building Labor Market Statistics with Administrative Labor Market Data Authors: Tomaz Cajner, Leland D. Crane, Ryan A. Decker, Adrian Hamins-Puertolas, Christopher Kurz, Daniel Villar Vallenas Discussant: John C. Haltiwanger Video: https://www.youtube.com/watch?v=HSCV6-Vfc4U&t=1446s ## Talk (00:24:06 – 00:41:40) [00:24:06] >> Thanks, Steve. Uh Ryan Decker from the Federal Reserve is our next speaker. [00:25:07] All right. That took a long time because I was slow in sending in my slides. So, sorry about that. Uh thanks to the organizers for having us. We're very happy to be here. This is joint work with this whole team here from the board, most of whom are here. Daniel [00:25:21] couldn't be here today. And today I'm going to talk about some lessons that we've learned from about 10 years of working with alternative or rather uh private sector administrative data. [00:25:32] We're not calling them alternative because Matthew doesn't like that. So uh private sector administrative labor market data uh and in our case from the payroll processing company ADP. [00:25:42] And I'm going to talk not only about building these data sets but also about using them because uh we're central bank staffers and we do we have to do macro stuff in real time. The usual disclaimer applies here which is that my comments today are just my own views and not [00:25:56] those of the Fed. This chart here will be familiar to you. We just saw it. The red line here is the uh response rate in uh CES. And I think it is fair to say uh that a lot of our economic statistics [00:26:09] have been under a bit of stress uh increasingly over time. Response rates are one way that you see that we've also had some large uh well large relative to CES benchmark revisions. They're not so large relative to some products GDP and stuff, but they've uh certainly [00:26:24] increased uh for the last few years. this year looks like it's actually not going to probably revise but um you know this has caught some attention and some heat has come on on statistical production at the same time in recent years we've seen kind of an explosion in [00:26:38] the availability of private sector data uh some of it was before the pandemic the pandemic seems to have jumpstarted a lot of new stuff and so the question and and what people are thinking in this conference is uh you know can we address some of the challenges in the official [00:26:51] data with alternative data this is a lot of sort of boilerplate Probably most people have thought about this, but uh there are trade-offs associated with both official data and alternative data. When we think about official data, you know, in terms of the [00:27:05] positive, we have super high quality uh transparent methodologies. We have representative samples that are constructed scientifically. We have really long time series. Uh in some cases, particularly CS, we have really high quality benchmark data. CS is [00:27:19] probably the best one. Um but there are also limitations. There's various sources of error. It's hard to measure birth and death as Steve was saying. In fact, in a business survey, you basically can't measure it and and CES is not alone in that uh problem. And you don't observe truth as often as you [00:27:33] would like to. CES observes truth once a year, March uh in the benchmark uh revisions. You might like to have it uh more often than that. [00:27:41] So, what about private data? Uh it has its own set of trade-offs. Some of these sources have very large samples. I would actually say I I should probably most of them probably don't have super large samples but uh the one we worked with ADP had very large samples. They might be complimentary to the samples we're [00:27:55] seeing in the official statistics. Uh you get some improvements in times in terms of timeliness. You get the data quicker, might be higher frequency, might be more granular in terms of industry or geography. Uh but there are downsides here too. There's selection [00:28:09] problems everywhere you turn in these data sets. some of which you can address if it's observable and you have good weights like from the QCW, but some of it you just simply can't. Certain types of businesses choose to use a payroll provider. They choose to use a particular payroll provider and you [00:28:22] can't really observe why they do that. Uh you really never are going to observe the population truth in most of these data sets, at least in the labor market ones that we tend to uh work with. You might have uh lots of other problems. Uh short time series history. How many business cycles do you have? Can you [00:28:37] seasonally adjust with the data that you have in hand? So there's trade-offs here, but we've certainly seen a lot of new sources. This paper that we wrote is about our decade with uh the ADP data. [00:28:48] As I said, and uh we produced the data and then we and our colleagues uh used the data uh over over the many years that we were doing this. ADP is the country's maybe it's the world's largest payroll processing company. They cover [00:29:01] about 1/5if of US employment. That's comparable to the size of the CES sample. It's a very big sample of uh businesses. and we had anonymized microdata access. We received new data weekly with only about a week's delay from the reference period we were [00:29:16] looking at. So that was very very nice uh very timely. And we had a couple of different employment measures in these data. One was called active employment. [00:29:26] This basically counts the number of employees that are in the payroll database whether they're getting a paycheck or not in this period. And we had what was called paid employment which is counting paychecks. Okay. So this is two ways of measuring things. [00:29:38] and uh at different times different ones are useful. We had some industry detail and so on. And what we did was we built payroll indexes. We also built wage indexes. We didn't have time to cover a lot of the wage stuff in this paper. Uh there's there's uh another team that has [00:29:52] some overlap with us uh that was doing that. I'll just briefly cover it, but it was a very nice project too. And as I'll show you or at least try to convince you, we found these data really really useful for helping us track underlying labor market momentum. sometimes [00:30:06] predicting revisions to CES data uh and doing some high frequency stuff. [00:30:11] I don't have any equations but I do have a couple of slides on uh methodological things here and uh I'll be focusing on the employment measures not the wage measures I just mentioned and what I want to convey to you is that if you're going to build a statistic that you want [00:30:25] to use as an economic statistic but you're doing it in data that are just not made for that you face lots and lots of methodological choices and you have to make choices and uh it's not always obvious what the right answer is and then sometimes as if you're a data user [00:30:39] you find that you might want to change your mind about some of those things. Uh and so here I just hope to sort of uh illustrate that there's lots of choices to make. So, uh first of all, birth and death. As Steve said, uh we couldn't observe birth and death either. You [00:30:54] could show up in the sample because you're a 10-year-old firm just starting to use this company for payrolls, or you could show up in there when you're brand new, but we had no way of knowing. What we ended up doing was something kind of similar to CES. We calculated a continuer growth rate, kind of like the weighted link relative in the CES [00:31:08] product. uh and then we're going to clean up uh the birth desk stuff later. [00:31:11] I'll come back to that. Uh we used QCW which is critical as basically a set of sampling weights so that we could correct representiveness in terms of observables. ADP actually has pretty good coverage across the size distribution across industries. Uh even [00:31:25] at the small end it has an a huge number of businesses but still we did find that when we hit it with these sampling weights uh things improved. Of course we can't do anything about selection on unobservables and so on. We're going to hope that benchmarking will clean that [00:31:39] sort of thing up over time and and keep us on track. [00:31:43] Uh we're going to chain the growth rates into a weekly index. Uh and then we're going to seasonally adjust. Here you can make some choices. Again, you could seasonally adjust at the aggregate level or at the detailed industry. The CES kind of does on the more detailed side. [00:31:55] Uh we ended up uh seasonally adjusting at the aggregate level. We did it in the weekly data which is a little tricky and uh we did concurrent seasonals uh in the end. [00:32:06] So once we had a seasonally adjusted weekly payroll index, we had aggregate series, we had some sector level series that needs to be translated into jobs and this is a problem you face anywhere. [00:32:17] Nobody's covering the whole economy. So you've got to somehow map it to the whole economy and uh so we're going to benchmark it or benchmark it to the CES which is benchmarked to QCW plus a couple other little things. Um so we did this uh every year. We sometimes toyed [00:32:31] around with quarterly. never really got it uh implemented. So, we benchmarked once a year at about the same time as CES typically. Uh how to deal with uh birth and death. I mentioned that we focus on continuing establishments uh [00:32:45] and looking at their growth rate. We can benchmark in history to stay on track with the full employment universe, but what do you do going forward? The CES uses the net birth death model. It's a very nice model. It's complicated. Uh it would be a lot of work for us and we're [00:32:59] a small team. We used something we called a forward benchmark. Uh this is not so different from things that have been done in official data. In fact, I think prior to the mid-200s, it's about what CES was doing. I think they called it bias adjustment. Uh I think the retail trade survey still basically does [00:33:13] something like this. And what we do is we look at how well we did relative to the last benchmark and then we project forward that error or some modeling of that error into the future so that our real-time numbers coming out have this adjustment built into them and uh we [00:33:27] sort of bank on that to deal with not only the birth death problem but also other issues that we might be missing with our data. [00:33:35] I mentioned that we had two employment variables uh we had this active and this paid one. paid employment, which counts paychecks, is actually conceptually very close to what the CES is doing. Active is not quite, but actually in practice, [00:33:49] what we found is that active was a lot easier to work with. Uh, it wasn't as noisy and it helped us track CES a little better. So, most of the time we actually used active employment, though there were times when you used paid, and I'll I'll show that to you. Uh, and I wrote other housekeeping. All these [00:34:02] things occupied hours and hours. Uh, but you have to think about uh lining up paychecks across bi-weekly, weekly, monthly payers. the bi-weeklys are on odd or even uh weeks and so on. Uh we would snap things to Saturdays. We had some adjustments. Uh we talk about all [00:34:16] this stuff a little bit more in the paper. [00:34:20] So we built these pay payroll measures uh and they can give us weekly or monthly net job gains for the private sector and for broad sectors. So how do we decide if they're any good? Well, here I want to emphasize I think perhaps [00:34:33] an underappreciated thing about even CES. As I mentioned, CES only observes truth in March. Even final revised CES data, if you go look at those monthly wiggles, those wiggles still reflect the [00:34:47] sampling error. They're basically the third print estimate. Uh they've got birth death in them. So, even the CES data at monthly frequency, you probably don't want to treat them as total truth. [00:34:57] And so, tracking those closely, there's a lot of good reasons you want to do that. And we wanted to do that. Uh but maybe a more important test is just can we hit the benchmark revisions and can we see the benchmark employment level coming in real time and so that was an early test that we're interested in. [00:35:12] I'll show you some stuff on that but it is true also we're very interested can we identify in real time uh periods of CES mismeasurement can we predict the monthly patterns and can we follow rapidly changing economic developments so on the first one benchmark revisions. [00:35:25] So in this chart, the red bars are showing you the revision to our ADPR FRB payroll index. If it's positive, that means it had been running too low and it had to revise up. Okay? And the blue bars are showing the same thing for the CES. You can see there the recent uh [00:35:40] three uh bigger revisions. And uh you know, we followed these over time. We have some papers where we actually thought let's combine the signal from these two with a Calman filter. So we have some papers where we ran a Calman filter through these. And what we found from the Calman filter was [00:35:55] it basically told us that if you take a 60/40 weighted average, 60% CES, 40% ADPFRB, you do pretty well at tracking sort of an underlying labor market state. So the yellow dots there are the 60/40 weighted average of these two [00:36:09] series. And you'll notice a few things. The ADPFRB revisions are bigger than CES. It's a very good product. It'd be hard to beat it, but they're not much bigger. and they sometimes go in the opposite direction which is actually helpful and in these recent large [00:36:22] revision years in fact that is what happened. You'll also notice that that 60/40 does pretty well and the nice thing about the 60/40 is that we can see that in real time well before it's obvious from other sources even like QCW that there's going to be a large benchmark revision. So we were fairly [00:36:37] happy uh with that. Here's an example of a year in which ADPFRB was pretty helpful. So during the summer of 2021, we had been recovering from the pandemic recession and the blue dots show you [00:36:52] first prints of CES. And what it looked like in those data was that we were running out of steam a little bit. [00:36:58] Things were slowing down. But our ADPFRB data, which don't really revise on like a monthly cadence, only at the benchmarks, they were much hotter than those first prints of CES. And what we found over time is that CES revised up [00:37:09] toward ADF ADPFRB in that year. And so we were very pleased with that. Now part of that was uh because of first to third prints and so it was revising up even by the third print sometimes. Um I can even [00:37:23] show you another episode which was similar. This is from 24 to 2024 to 2025 and this time the CES numbers were running really hot. We probably remember this period ADP was running cooler and in the end CES revised down uh towards us. So these can help you you know in [00:37:37] real time if you're trying to do macro analysis and so on. Um and then even at monthly frequency uh we've done some analysis that's in the paper where even just running regressions on just the monthly wiggles we have a pretty good correlation uh with the CES. [00:37:52] Okay. So here's an example of a very high frequency event. This is the pandemic. Um early on in the pandemic it became clear to us that our paid employment measure was going to be more useful than our active employment measure. Why? Because it's going to pick [00:38:06] up like temporary layoffs and stuff. and we knew there was going to be an enormous amount of temporary layoffs. [00:38:11] The purple line here is showing you what that paid measure do did. And I'm showing you the viewpoint from early May of 2020 if you dare remember or think about that time. So, we're receiving the April jobs report in early May. And here [00:38:25] is what we could see. As I said, the purple line is paid employment. The red line is active. It doesn't fall so much because it's keeping people on the in the databases. [00:38:34] The blue triangle is the consensus market forecast for CES on the eve of release and they actually did pretty well. This was a couple months into the pandemic. Uh they're using initial claims data and other things to forecast. But you can see in the hollow [00:38:47] blue circle what CES did which is uh it did about what we expected the markets expected it to. So but it was impressive. Uh but let's go forward one month here to early June when we're getting the May jobs report. Now, what [00:39:01] had happened was even when that April report came out, we could already see if I go back and show you, we could already see in the purple line that paid employment was leveling off. We knew that when the April report came out, it was leveling off in the back half of the [00:39:16] month after the reference period. So, here we're going to get the May jobs report. The the triangle tells you the markets were expecting several million more job losses that that month. and they were based on not only claims data but even some of the other private [00:39:29] sector data out there at that point. We saw in the payday at ADPFRB data that things were stabilizing and indeed that is uh that's what happened in the CES. [00:39:38] So you can see this was tremendously useful at a time when we were desperate for information at high frequency. [00:39:44] The last thing I'm going to show is our wage measure. I would have loved to spend more time on this uh but we kind of focused on the employment here. But with the ADP data we can also c uh measure lots of different types of wage concepts. The blue and red lines here are some 12-month wage growth uh [00:39:59] measures from ADPFRB. What we can do is at the person level, we can measure your 12-month wage growth and then we can take a median. So, we can abstract from composition effects that as a lot of folks probably know in the average hourly earnings were a very very big deal. We don't have that [00:40:13] problem here. And it's sort of similar to the Atlanta wage growth tracker, but a much much larger sample. So, the wage data were also uh tremendously useful to us. [00:40:23] So, let me just wrap up with some uh lessons learned, I guess. Uh the first is hopefully I've convinced you that uh these private payroll processor data were really useful for measuring both high and low frequency employment changes as well as wages, and they're a [00:40:37] nice complement to official data. They didn't do better than official data, but they did pretty well. Uh producing statistics from these raw data is pretty difficult. It requires careful methodological design and judgments. And I think the thing we learned here is [00:40:49] what you'd really like is for uh experts to have their hands on the data. Um you can get a lot of mileage if you have high quality benchmarks and stuff but the data are most useful in micro data form uh or close equivalent uh given [00:41:03] sort of the complicated methodological choices that we've been making. That being said in the paper we write some more speculative ideas about how technology might enable more flexible data sharing arrangements. So if these companies are very nervous about sharing [00:41:17] their data, you could imagine ways to get in there, get cell level data, bring them out in a confidential way, uh, and, uh, maybe put some of these things together. So you can read the paper and and see what we thought about that. And I'll stop there. ## Discussion — John C. Haltiwanger (00:41:40 – 00:53:43) *Shared discussion for Session 1: Current Employment Statistics; the discussant covers every paper in the session.* [00:41:40] Thank you. Thank you, Ryan. Our discussion is John Halawanganger. [00:42:30] Thank you. So it's it's great to be here. Very nice conference. It's it's a real pleasure to get a chance to talk about these these two papers. These were just two two excellent uh presentations. [00:42:40] Actually these are outstanding papers. I highly recommend reading you say all all the details what what comes across. I won't give given limited time. I won't spend time kind of resummarizing. I think the presentations were great. uh if it wasn't already clear to you from the presentations, it will be when you [00:42:54] read the papers. The expertise underlying these papers is just enormous. The the staff at BLS have been thinking about these issues for for many years have great insights and therb team u essentially took on the ADP and said [00:43:08] we we need to go worry about everything the stats are worried about and and do the best we can. So I'm going to talk more about I'll call it common threads but also sort of complements and substitute kind of issues about when does the ADP data really kick in and [00:43:22] potentially help us when maybe not so much and then and then related to Matthew's opening remarks okay how do we how do we go forward from here not not that I know the answer but but I want to talk about that a little bit so [00:43:35] if if it wasn't clear to you already I mean what's striking about both papers and indeed we we kind of know this anybody's who studied this issue the ben benchmark data is essential. You can't get there from here without really high quality benchmark data. So we we've [00:43:50] known about this for a long time and indeed the the CES is is is a beneficiary of of one of the most timely benchmarks out there. We know for the BA folks in the audience obviously benchmark revisions for GDP uh come uh [00:44:04] at a much slower pace and so and and obviously the ADP paper relied very heavily literally on like every step of the way. Okay, they they know they had a non-representative sample. They had to work worry about weights. They obviously [00:44:17] wanted to do the benchmark revision and then they also had the birth depth problem. Basically the same problem uh that that the CS faces and and and and they they used a similar but perhaps not quite as sophisticated uh method. So so one of the areas I'm going to push on a [00:44:32] little bit is if if given that the benchmarks are so critical, what can we do to improve the benchmarks both in terms of quality and timeliness? I think that's that's a message that emerges uh from from this line of work. So So one [00:44:46] topic I want to talk briefly about and I know the folks in BLS are like experts on this because they've been talking about this for years is why not quarterly benchmarking? Indeed the QCW has the has the potential for uh quarterly benchmarking and this has been studied by BLS staff amongst others and [00:45:00] and I'll talk about what I know about the the limitations why this isn't being done now but we know we know lots of people are using it. uh indeed folks are using the releases of the quarterly QCW all the time to predict the the annual benchmark. So it's it's actively used by [00:45:15] the I'll say uh market participants. It's also being used by BEA in for example in the national company project counts. So it's not as though people aren't using it already. You could say well why isn't BLS doing this? Well I I I as best as I understand it one is [00:45:29] there are known quality differences across the quarters. the first quarter data is of higher quality. But my my my understanding is that's probably a resource issue which is in in in principle if you could devote the same amount of resources that you were able [00:45:43] to devote to the first quarter you could make the second third and fourth quarter as high quality as well and that may be really valuable given what we just heard from these two uh excellent papers. I also heard that and indeed the the BLS paper nicely talks about this [00:45:56] benchmarking is not cheap right to actually do that benchmarking revision takes time and resources. So you could say these are both resource issues and and obviously the folks in this room we are in a resource constrained environment. So how the question is how to get there from here but still given [00:46:09] the importance of this benchmark I think a priority ought to be put on this. So one idea and it kind of came up a little bit in some of the discussion and in the papers we are in a technological revolution going on right now in terms of our ability to process [00:46:22] data in terms of modernizing processing data and AI. The question is can we literally reinvent the systems the old the old SAS and and and other kinds of systems that are that are sitting inside [00:46:34] the agencies and move to Python and AI. Uh is that the place so that we can for example process our administrative data in a in a much more timely fashion. It's also the case and Ryan kind of alluded at this and I'm going to draw broad draw [00:46:48] out a point that he that he raised which is you know amongst other challenges the ADP data has is all the missing industry codes. It's it's non-trivial and the assumption that they had to make was really strong assumption in terms of you [00:47:02] know inside cells missing at random and we and we and we don't think that's probably true at all. It's that's that by the way CS has other missing at random problems as well. So, so you know amongst other things even though uh [00:47:13] clearly the inter entry into ADP is is not um is not true business entry but but there was some hints in the paper that there's some connection between some of the turnover of of entry associated with business entry and and [00:47:27] exit and that's that's a source of non-randomness. [00:47:31] So as Ryan uh suggested at the end the the really the way to get there from here and the question is how do again how do we as a community get there from here if it'd be great you mean you could overcome this aspect of the limitations [00:47:44] of ADP if actually the ADP data could be integrated with the micro uh QCW that's a that's a that's a objective we we should uh we should push towards now the other thing we heard again a common [00:47:57] theme is births and deaths are are hard and they're important right? They play they play a big role in in uh in the nature of the revisions and and they're really hard particularly at turning points uh uh to get right and and and [00:48:11] this is just a reminder maybe was already clear from Brian's very nice presentation. [00:48:16] The the payroll data doesn't help you really at all on this. Okay, you're not you're not getting any help uh uh here. [00:48:21] So the question is can the model be improved? Here's where I think we ought to be thinking about what are really I'll say highfrequency timely indicators that that administrative data that we might be able to tap into to improve our [00:48:36] tracking of births and deaths. So, so one idea I'm a little biased here because I hand in this work at at census the business formation statistics is is basically real- time processing of new new business applications. That flow of [00:48:48] data is weekly almost in real time. we ought to think about whether that data could be potentially used in this context. I've actually also wondered whether whether BLS has ever explored using AI UI claims because that's actually sitting there uh uh and maybe [00:49:02] there's a many papers cited in the in the in the long bibliography that's already been done. I also think and this is related a little bit to the modernization AI point is we know we know that that the tax data both the [00:49:16] payroll tax data and the UI tax data it it's coming in u much more rapidly than than by the time it shows up in the QCW or at census in the business register question is can those systems be modernized so that so [00:49:31] that one is literally we get more timely processing of the registers but but but also early reads so so it may not it may be that there's signal in a in a very early read out of the administrative tax data. Can we can we tap into that? [00:49:45] So here's a place where I think the ADP data really is an advantage. So one thing we know and indeed has been a theme is non uh response rates have been falling and so therefore the all the assumptions that we're making about [00:49:58] non-response bias have become more and more important o over time and I and I think this idea that there's inside cells there's missing at random is just wrong. I think we already know that. [00:50:09] I'll say in work actually that I did with Steve Davis and Jason Faberman and Ian Rucker, we we we had a very a project that matched up the micro uh Jolts data with the BLS LDB inside a project within BLS. We found very [00:50:24] systematic evidence that it wasn't just businesses that as as talked about in the BLS paper that are shutting down that don't that don't uh file reports. [00:50:33] Actually, businesses that are contracting rapidly aren't filing reports. How could we tell? because we actually had the QCW data that we could match up to the to the Joltz data. So I I know BLS is thinking about extending ways of of handling the kind of the non [00:50:48] response bias and and and and I applaud that but but this is a big problem and and and unfortunately getting worse as the response rates have declined here in principle I think that the ADP data [00:51:01] avoids mostly avoids this problem. Although that'd be interesting to hear from that team is that well maybe not completely. Why? because it may be that you don't respond to the BLS survey, but you're still making your payroll is the point. [00:51:12] So, let let me bring this this to a close with a like the really big picture question on the on the table and it's related to Matthew's uh opening remarks. [00:51:23] So, there's a vision that what the stat agencies should particularly be doing in terms of surveys whatever and uh is the benchmark data of administrative and then and then benchmark like surveys. the economic [00:51:37] census, the annual surveys that provide the key benchmarks and that we should be tapping into this naturally occurring transactions data for the high frequency. Could we do that for the PFIs? And I think that's the big question on the on the table and the [00:51:51] question you question we should ask what what do we think about I don't have the answer to that question now but do we think how close are we with the payroll provider data and and the great thing is the ADP uh paper out of the board gives us a a preliminary answer out of this [00:52:04] and what's the preliminary answer we're not ready yet okay that that's not a criticism at all as as Ryan said this is a great supplement to the official data at this point it's it's not a substitute okay we're we're not there at [00:52:19] does that mean we can't get there from here? No. I think actually that's exactly where we ought to be pushing. In some respects, some of Ryan's remarks and so on uh suggest even though ADP is huge given its non-representativeness and that there are other payroll [00:52:32] providers, we we're hear about some of those at this conference that cover some areas like young and small businesses to a greater extent than than does ADP which will help us on some of the dimensions um that they talked about and I talked about. So, so what would I hope [00:52:47] that we as a community do that somehow or another we find the resources to do a large scale demonstration project tapping into multiple uh data providers that ought to be our objective and ideally obviously not only ideally it needs to be with the collaboration in in [00:53:02] this case particularly BLS for this to work and really ideally it needs to be collaboration with BLS in a way that the that this naturally occurring transaction data can be as as we've talked about before is integrated with a [00:53:15] micro QCW. That's the gold standard we ought to be pushing for. The question is how to get there from here. I I know there are there are huge issues about access issues and governance issues and so on for tapping into this kind of data. I I think we ought to partly split [00:53:29] those issues apart. First, we just ought to go see if we could actually tap into this data and integrate it with this fantastic benchmark data, could we could we overcome many of the issues uh that was were nicely talked about here. I'll ## Q&A (00:53:43 – 01:08:20) *Shared floor discussion for Session 1: Current Employment Statistics.* [00:53:43] stop there. Thank you. Um, thanks John. Thanks John. Thanks all [00:54:01] the speakers for being on time. So we have plenty of time for comments and questions. I think I suggest we collect some questions and comments. And David Wilcox, do you want to kick us off? [00:54:20] I had the great blessing of a durable long ve long marriage. My wife reminds me that sometimes it's important to ask the basic questions and state the [00:54:32] obvious things. It feels to me like uh an obvious thing that hasn't been stated is it was implicit in Ryan's comments. [00:54:43] Your discussion was in the past tense. ADP is no longer providing those data to the Federal Reserve Board. [00:54:54] Um so in addition to all the other lessons that we've learned which are tremendously valuable, I think I What I'd like to hear some [00:55:06] recognition of what we take away from that experience. The Federal Reserve Board effort was a live ammunition exercise. We were putting this information in front of the open market committee and allowing them to make [00:55:20] policy decisions because we decided that the product the blended product was high enough quality to warrant that and suddenly it was cut off. What happens if [00:55:33] What happens if >> that that was [00:55:50] >> okay? >> Thanks. [00:55:53] >> Could you could you say who could you announce your name for? [00:55:55] >> Sure. Sure. My name is Michael Steer, professor assistant professor at the University of Toronto, collaborator of the many of these uh um administrative uh private sector data projects. I want [00:56:09] to just raise room. There's another angle here, right? We talked about administrative private sector data. The other angle is improving the administrative data that the government gets by requirement. And uh my former [00:56:23] collaborator Peter Ganon is hosting a conference in November about the UK's experience where the government reformed payroll reporting requirements to require companies to report more regularly [00:56:36] uh to the government to their payroll. So the idea would be the analogy here would be rather than making do with the best we can of the QCW to benchmark all [00:56:48] these private sector data what about um improving the QCW of course that has its own challenges and uh changing government policy but I think it's worth thinking as we think about modernizing [00:57:03] our employment measures about both things because right now we're doing what we can with what we have. There may be other angles and other countries that [00:57:12] we can look to for um other models here. >> Thanks. [00:57:33] >> Hello, I'm Hijuan from the Federal Reserve Board. So I have a question about the you know non-random missing observation and that potential bias to the seniors and the joy. So as John mentioned and also you know mentioned by [00:57:46] Ryan um that certainly this missing observations are not responses in the you know established survey um is not random. So there's evidence for it and certainly um it can be a you know challenge you know the payroll data in a [00:58:01] sense that that actually means the ADP data can be actually more um timely in detecting recessions or expansion especially there is a you know pattern of bias that you know like underperformers tend to not respond to [00:58:15] the surveys you know so um but I was wondering you know but additional information to bear is always help us to identify the patterns of bias arising from nonresponses so I was wondering whether you have a chance to study this [00:58:29] or you know think to think about the strategy of you know about this bias at is the direction of magnitude based on the AD. [00:58:39] >> Thanks, John. Stable houses. >> Yeah. Yeah, it's on, John. I think it's [00:58:54] on. >> Uh, so Brian, show us some examples of periods in history where FRBA did a jobs suggesting CDS be pulled back in that direction. I guess the question is are [00:59:09] there other periods in history where it went the other direction and if so what would we say about different types of uh I guess points of business okay I will use uh u organizers [00:59:34] privilege so I I I think David's question and and Michael's uh Michael's are actually related. So if you if you continue to have alternative data feeds [00:59:47] as ad hoc measures, this kind of thing can happen and and uh these are basically dependent on contractual relationships. other hand if you if you take the data not alternative data but it's the data and start feeding it in to [01:00:01] the official process some of it could go off to the states for benefits processing which is uh some of it could go to the BLS for uh statistical processing those could be coordinated the data become for the payroll [01:00:15] providers or the individual payrolls of firms in in this modern way but it would the process would be integrated with the data collection And that that's a different infrastructure than sort of ad hoc relationships of purchasing or [01:00:29] cajoling data providers. So we we have a whole panel on this tomorrow. So I hope we take this we can think about this and discuss this further. Did the if there are not further questions like I did the uh presenters want to have a chance to [01:00:43] say something? Uh >> uh is it >> lean? Could you introduce yourself just [01:00:58] for the record? >> Yes. Just just say your name for the just >> Yes. Thank you. [01:01:05] >> No. So John's questions for the AP side. It's like non response is not a problem. [01:01:16] universe once you're in. [01:01:23] We always had questions whether it switch yourself anymore. So it's something we kind of looked at trying to find patterns the [01:01:36] same way. Okay. Okay. Okay. [01:01:45] And then um to John's question about revisions. So again, I think we don't have much of a problem [01:01:56] with first and third as we get monthly semers in, but we don't have a third significant third month happening the same way CBS does [01:02:10] because folks just don't get the reports on time. So for those kinds of third revisions, ADP was probably has significantly smaller revisions. That said, we saw the benchmark figures and [01:02:23] there were times when um had benchmarks did so we don't like cherry picking um particular cases too much but that's [01:02:37] >> I was interested about direction and size. [01:02:44] like are you pulling back towards CVS or uh are you suggesting that the C are you pushing against CDS is going to get revised? [01:02:55] >> Um I mean we might need to look at the data some more but again we're not you know ADP wasn't revising much so CBS will sometimes be revising occasionally they would be revising away from ADP but [01:03:08] >> I mean there were years when uh ADPFRB benchmark revision was uh bigger I mean many years in which it was bigger than CVS and there were years where it was bigger than CDS and in the same direction. So it was this so if you had [01:03:21] taken too much signal in that year it would have aired. I mean conveniently in these really big CS benchmark revisions that had done really well. So when the states were high it was helpful but there were other times in 2017 2018 it was on that chart I showed when it [01:03:35] wasn't so good. So, uh, you know, macro forecasters, which all of our great labor forecasters are actually here right now, have to decide how to weigh these different signals and and what to do about it. I'll make one comment on uh [01:03:49] this point about the holdup problem. So, this I'm just going to plug a statistic we produce at the Fed called the Industrial Production and Capacity utilization statistic. And this is a statistic that for has been going for over a hundred years, but for that whole time it has been a blended data product. [01:04:03] And so we have lots and lots of of private sector sources from all over the place uh that feed into that. And over many years we have lost many many uh sources and I think that the cost of [01:04:17] losing a source it's partly dependent on obviously how important that source is. [01:04:21] If you were using something like ADP which is so huge uh and if you were very dependent on it it would be very costly to lose it. But at the same time I think the cost of losing these sources depends a little bit on how good your benchmarks are. how soon can you catch it and can you replace it? Can you come up with [01:04:35] some sort of a workaround? And um I think the BEA has lots of experience with this in the Nippas. They've had lots and lots of sources over the years, but I think you've come gone. And so it's a very tough problem. Uh and how you design a system, you know, you want to keep it in mind, but I think it's [01:04:49] also a surmountable problem. Katherine, >> I just had a a slightly different thing to ask about, which is a lot of times [01:05:01] when people talk about augmenting the official statistics with other data, they're interested in being able to produce more disagregated statistics. And when I, if I'm [01:05:16] remembering your paper correctly, you said you just tried to wake things up at the national level. you didn't try to reweight the geographic bubble um and and it didn't look like you were interested in producing geographically [01:05:30] disagregated statistics, but I'm wondering what thoughts you had about how incorporating AP data in conjunction with the CES data might help with that [01:05:40] or whether it wouldn't be that helpful. >> Yeah. So uh yes uh we we didn't do waiting by region. We did over the years many times look at sort of regional patterns uh like if there was a hurricane or something. So we did [01:05:54] sometimes do that. Um one challenge in probably a lot of these data sets is that sometimes you'd like to imagine if you know what kind of business unit you're dealing with. Are you dealing with an estate or a firm? But as we wrote in the paper, it wasn't always so [01:06:07] clear. We treated these businesses as establishments. Some of them might have been firms. And if you have uh firms masquerading as establishments and that go try to do regional stuff, you're going to be misled. And so a very large multi-unit firm, if it's looking like [01:06:21] it's all located in one place, it can really throw things off. So, uh the data was certainly that sort of thing. Um but it again comes down to these tricky like paying very close attention to [01:06:34] measurement choices. I'll just comment real quick on the non-response issue and data that are not missing at random and when the respond is related to what we're trying to [01:06:48] measure that really problematic and so what we're trying to do is look at are there other things that you could observe that we could make some adjustments for so uh I know John mentioned in relation to birth depth which we actually did look at back in [01:07:02] 2013 and uh I think it worked okay Um this last round we really tried to tie it to CS CS data but um we've been looking at um this is this is really [01:07:15] messy but um one looking at count's data linked like this but um if a business that's not responding lighting [01:07:29] up a lot of people that probably tells us something about what's happening there. So looking at stuff like that we're able to some of these Friday data um with CS survey data we might be able to make that as well as [01:07:44] we want to do I'm sure about work but some of them do >> uh thanks so I want to thank the the uh [01:07:57] authors for these papers these papers were commissioned for this uh for this conference and it got us off to a great start by setting out the issues and showing showing how much progress has been made but and I hope it sets up the [01:08:11] the foundation for further progress. We'll take a 20-minut break and return promptly at 2:40. Thank you.