Transcript of the talk, from the video's captions. Auto-generated: speaker names in particular are unreliable. ======================================================================== # Recent Developments in Current Employment Statistics Authors: Christopher Manning, Steven Mance Discussant: John C. Haltiwanger Video: https://www.youtube.com/watch?v=HSCV6-Vfc4U&t=235s ## Talk (00:03:55 – 00:24:06) [00:03:59] So, and our first presenter is uh Christopher Manning of um of uh of [00:04:13] BLS who will talk about recent developments in uh in this area of BLS. [00:05:03] Okay, so I am not Chris Manning, although with a similar enough last name, I sometimes get his emails with the autocomplete. So, uh, Chris Manning is our, uh, CES program manager and I work on statistical methods. [00:05:17] Uh, usual disclaimer, um, any opinions are, uh, those are the authors and do not constitute, uh, BLS policy. [00:05:25] So, this is a pretty wide ranging topic. I'll keep it to the 20 minute limit. Um, first I'll do a very high level 30,000 foot overview of CES. Uh, most of you probably have plenty of familiarity [00:05:37] already. Uh, then I'll go into four areas that could be considered either uh, challenges or opportunities for CES and how we're addressing it. So, uh, first I'll talk a little bit about late sample uh, monthly revisions and unit [00:05:52] non-response. Uh, then I'll briefly discuss seasonal adjustments role in the monthly revisions for CES. Uh then on to my favorite topic about uh the birth death model uh and business openings and [00:06:05] closings and then I'll wrap it up with a little bit about uh small area estimation and uh our uh investigations into alternative data which is fun and exciting. [00:06:18] So this will be the very very quick overview of CES. Um as most of you know we produce uh data on non-farm payroll employment hours and earnings by data and uh by industry and geography. Um the [00:06:32] national data are released about uh 3 weeks after the week that includes the 12th of the month which is our reference pay period. The state and metropolitan data come out about 2 weeks after that. [00:06:43] We are a survey of about 119,000 businesses and government agencies representing uh about 622,000 individual work sites. And this makes us among the timeliest timeliest principal federal [00:06:58] economic indicators. It's closely watched and um yeah, usually out first Friday at 8:30 a.m. [00:07:06] Uh very quick look at our estimator. uh our current all our current month all employees estimate ae hat subt uh basically it's recursive we take the uh [00:07:19] prior month's employment estimated employment level this gray term is our um atypical or self-representing employment you can pretty much ignore that um then we multiply that by this ratio of weighted matched employment [00:07:33] from the current month divide by the employment by the prior month so essentially what we're doing is we're um uh taking an estimate of relative employment change from uh from our match sample and we are using uh only [00:07:47] establishments that reported positive employment in both months. [00:07:51] Uh so with this and also at the end of this we add on this uh B subt which is our birth death uh adjustment. So that's basically uh the amount that we have to uh adjust the estimates due to our expected error from our difficulties [00:08:06] capturing uh new businesses in real time and business closures. Uh so with this there's a essentially a um an implicit imputation for non non-respondents or late respondents within each estimating cell which is done usually at the [00:08:20] detailed snakes and in some cases region level. Um and within this we usually have a uh essentially a missing completely at random assumption which is a pretty strong assumption. Uh but there are some cases where we make uh imputations for known influential units [00:08:34] that are seasonal and uh late. So talking a little bit about late sample and revisions. Uh there are two types of revisions in CES monthly and annual. Um we revise the prior two months of data with each monthly uh [00:08:49] employment situation release. Uh this is because about onethird of our responding establishments uh report too late to be used for our initial estimates. Uh we'll also incorporate some other processing changes as well as resubmitted and [00:09:02] corrected data um if there's suspicious data that um we flag at our initial estimate. [00:09:08] Those those uh businesses will be reconted. So they'll either be corrected or um confirmed. Um we also update our seasonal adjustment factors for the prior two months. And then annually we [00:09:20] uh benchmark our levels to um uh administrative counts primarily from the quarterly census of employment and wages uh which covers um [00:09:32] uh pretty much all the 97% of non-farm jobs. So real quick look at our um revisions. So this is a 12-month trailing average. These are the monthly revisions. If I just plotted the [00:09:47] revisions themselves, it would be very very noisy. So, taking this 12-month average uh helps us see a little bit of a pattern. Uh the red line is the not seasonally adjusted uh revisions and blue line is for the seasonally adjusted revisions. And uh things that jump out [00:10:01] to me are that the seasonally adjusted revisions are a bit more cyclical. They were pretty negative around the great recession 2008 2009 and then um also both revisions were negative in 2020. [00:10:15] That was partially due to some uh methodological changes that we made between our initial and final March 2020 estimates. And then coming out of the uh the pandemic recovery essentially uh revisions were very positive. Um the [00:10:29] seasonally adjusted revisions were over 150,000 on average in 2021. And so that's that very high data point for the blue line. Uh since then they have uh been slightly negative over the last couple years. [00:10:43] Uh also I'll note that the magnitude of revisions um tends to be a little bit more cyclical for the seasonally adjusted data and um also that the last few years they've kind of moderated in size. So this is the absolute revisions I should have [00:10:58] said. um uh the last couple years since 2023 or so, they've been pretty comparable to uh where they were at say uh 2007 through 2013. [00:11:11] Uh so in terms of um data that's coming in between our initial and final estimates, uh we can look at our collection rates and the red line shows our final collection rate. So this will always be pretty high. It's typically [00:11:25] above 90%. Um and the denominator for our collection rate is the um we call this our active sample. So these are establishments that we're expecting to get. So this tends to be very high. Um so this is the proportion of our our match sample that we get for our final [00:11:39] estimate. And then the blue line at the bottom which is quite a bit noisier is our collection rate for our initial estimates. And you may notice that the uh these have diverged by quite a bit. [00:11:51] So the gap between our initial and final collection rate has grown quite a bit. [00:11:56] So, uh, David Volcox has, uh, noted that, um, uh, despite these, uh, these measures diverging by quite a lot, the, uh, size of the revisions, um, really [00:12:09] hasn't grown that much. So, um, it appears that we're getting a lot more information between our initial estimate and our final estimate. Um, but this may only be in terms of these establishment counts that we're reporting. So something that we looked [00:12:23] at as well to try to get another angle on this is uh weighted employments. [00:12:28] So uh the blue line here is the ratio of the first and third closing or the initial and the final uh collection rate and the uh red line is the ratio of our initial weighted matched employment [00:12:42] divided by our final weighted matched employment. So that's actually increased a little bit over time. Um so that's telling a very different story in terms of uh how much data that we're getting for our initial estimates. uh relative to our final estimates. Um and essentially what's happening is some [00:12:56] compositions in terms of the the sample that we're getting um for the first and third closing estimates. Um the late respondents are increasingly representing fewer weighted jobs per establishment. [00:13:08] Uh so what are we actually trying to do about revisions? Um one of the things that we've looked at is predictability. [00:13:14] Uh so if an outside analyst were to take our data and try to forecast some revisions um what would they what would they be able to do? So we did a uh little out of samp little uh forecasting exercise with uh the not seasonally [00:13:28] adjusted revisions for 100 individual industries. Uh these were mostly at the three-digit NAKES level and uh we did this over the course of about 10 years. [00:13:39] Um and most of these revisions at the so basically we fit uh these ARMA models with uh Trammo and just let it kind of do it do its thing automatically. So so maybe some more sophisticated approaches could have done a little bit better but the only industry that we saw that was [00:13:53] really predictable was local governments and this industry had very positive revisions in August pretty consistently. [00:14:01] So we developed um some regar based now casting imputations that uh since uh August of last year we've been providing to the BLS analysts that are actually making the estimates. Um we're not automatically putting these in. Uh [00:14:14] there's some reasons why but um uh this blue line shows I mean that looks like a very predictable series right um it's it's not white noise. Um those are the revisions for local government education [00:14:27] and the red line is um the out of what the revision would have been had we made these uh made these uh imputations. So that's still somewhat positive in August but generally quite a bit better. Uh [00:14:40] I'll note the collection rates were uh have held pretty high but uh our overall response rates have trended quite a bit lower as has been the case for most estab uh most surveys. Um these are the overall CES response rates. Uh they've [00:14:54] stabilized in the low 40s. Um and what are we doing about that? Um we uh one thing I'll mention is that we have a uh outside uh research firm that's done a non-response uh analysis with BLS. Uh [00:15:09] they've completed it. It should be posted publicly by the end of the calendar year. uh with BLS we've been doing some work uh with very preliminary results um looking at multiple imputation as well as um we're doing some uh treebased methods to uh develop [00:15:23] weight calibrations. We're having some success with it. Um there are also some uh interesting industry specific approaches um where we're using uh my favorite was a was in sports teams where we're looking at the sport that each one [00:15:37] of these establishments represents. And it turned out there was a month one time where baseball teams had a much higher response rate than uh hockey teams and ba basketball teams. So in April baseball's hiring basketball and hockey [00:15:52] is laying off. So that resulted in some non-response error for us. So had you made an adjustment for that you would have had a much better accur much better estimate. [00:16:01] Briefly I will touch on seasonal adjustments. [00:16:04] Uh so the revisions to the seasonally adjusted over the month change. Uh we can think of this as happening due to three uh three main reasons. There's underlying revisions to the not seasonal adjusted data. Um that affects the calculations of the seasonal adjustment [00:16:18] factor as well. Uh some publication decisions. Uh I mentioned earlier that we only update the seasonal adjustment factors for the prior two months. So this results in something of a splice point in terms of our our data series. [00:16:30] So our um our our what we call third closing over the month change. it's comparing two vintages of seasonal adjustments. Um and then forecast error. [00:16:40] So um we think that this is contributing to some of the cyclicality that we're seeing in the season seasonally adjusted revisions. And um uh so [00:16:52] in seasonal adjustment um we basically forecast extend our not seasonally adjusted series and then run a series of um uh X11 moving averages to decompose everything into the trend seasonal and [00:17:07] regular components. Um but we do these forecast extensions because uh otherwise you have to use these really bad um uh asymmetric moving averages. Um but it's hard to forecast when you have changes [00:17:20] in the economy which um has been happening a lot. Um so uh we are looking at uh seeing if there's some things that we can do to improve our forecast extensions. We directly adjust 148 [00:17:33] individual series. So we we're seeing a little bit of structure in kind of our forecast errors as well as the residuals for our models. And um we have a uh a project that will be kicking off this next fiscal year that starts in about a week or two. um trying to do some work [00:17:48] on that births and deaths and the birth death model. Um so the actual business births and deaths. Um so what we what we do is [00:17:59] we construct a um we have all this administrative data that we eventually benchmark everything to the quarterly census of employment and wages and um what we can do is well we know that we [00:18:13] can't sample and enroll and collect businesses in real time. it takes about a year uh by the time something has shown up on these UI tax records, they've been enrolled, collected, all that sort of sort of stuff. Um so we can't capture growth from these newest [00:18:26] businesses and there's a lot of job growth from very new businesses. [00:18:31] We also have difficulty capturing business closures. Um the assumption when we first implemented this was that if a business shuts down, they're not going to prioritize filling out their CES survey form, right? Um so what we [00:18:46] can do is basically assume that we follow that we don't capture any of these births or deaths and apply these CES processes to the administrative data. So we have um uh an administratively derived value that um [00:19:01] can tell us how much we would have had to adjust a CES-like estimate um under these under these assumptions. Um so that's the actual birth death value and then we forecast that. So um historically we did this with a seasonal [00:19:13] arma model which um was not at all responsive to uh to turning points. Uh one would imagine that business employment growth from business openings and closings may be somewhat cyclical. [00:19:26] Um in 2020 and 2021 we had a massive surge in reported business closures. So I mentioned that we assume that we don't get these but we got a lot of reported business closures. [00:19:38] And so uh many of these were temporary which was very fortunate for us and we included these in our that match sample term that I showed earlier. Um we treated them as as if they were matched even if they were reporting zero [00:19:51] employment. Um and we we did some await adjustments for that as well. Um but this tapered off pretty quickly. um it was sort of a temporary surge in this and um it it made it very clear that we [00:20:03] needed to have some sort of um you know approach that would work well kind of across the business cycle and so we started a research project looking at um a lot of different approaches to forecasting births and deaths. We [00:20:18] settled on a couple of regar style models. uh the the stuff that we really liked in our research was a little bit cludgy, but it gave us some similar results to a uh a more standard regar model uh that we've been using for the [00:20:30] last um last little bit. So, we um implemented these for our we call April through October 2024 postbenchmark yeah postbenchmark reestimates. Um we had a fairly large benchmark revision um with [00:20:45] that March 2025 benchmark. it would have been substantially larger had we not done this. Um, and then since January of this year, we've been using these models uh on an ongoing basis. So, real quick [00:20:58] look, I only have a couple minutes, but um this shows total private job growth over the last 13 years and compares it to the actual birth death values. So, the idea here is basically the birth death values are are really and this is [00:21:10] all from basically QCW data. The red line's been benchmarked, the blue line is derived from QCW. Um the the idea here is basically that birth death is really important to the overall trend. [00:21:21] Month-to-month variation, it matters, but it's far less important than uh change from continuing establishments. [00:21:28] And then this just shows a graph I really like of uh uh some of these birth death predictions that we made um that would have worked really well out of sample during the pandemic with a very simple linear relationship. [00:21:39] Um so we'll keep doing a few other things with birth death. Uh we're open to uh to improving some other areas uh of these models. Um what we have right now works well I think but um but there's definitely still some room for improvements. Uh we're also interested [00:21:53] in hearing more about um uh using some alternative data. Um which I guess brings me to the next section. [00:22:01] Uh so alternative data and related topic is uh small area estimation. Uh so Kyle Hood will talk a little bit more about this stuff later but uh BLS has been involved uh in a project working with BEA census and opportunity insights uh [00:22:15] working with a large payroll providers data and um seeing what we can do with that. Um there's been some good results in terms of expanding the um uh either the frequency of data that could be produced or uh the granularity at a [00:22:29] given level of accuracy. Um while this has been going on, BLS has been doing a lot of theoretical work uh people that work on um CES as well as our uh in our [00:22:39] survey methods research group um on uh uh methods for blending and augmenting uh probability data with uh non-probability data. So um uh first [00:22:53] I'll mention we have a a really good small area model that we are using in production for um our state or metropolitan area estimates that is um extensible with alternative data derived coariantss and then um these other [00:23:07] papers here are uh the actual methods for uh for the data blending. So um we have methods that we've developed for combining convenient samples with uh with our uh probability [00:23:20] data. Um we've looked at all sorts of different uh scenarios. Basically whether we know that there's overlaps between the convenience sample and the probability sample. Um whether we can do linkages, all that sort of great stuff. [00:23:33] So, um, and they are looking forward eagerly to, uh, to getting some of the, uh, some of the payroll provider data that hopefully they'll be able to work with. [00:23:45] And that is all I have. I just hit the 20 minutes. Great. Um, so again, here's our contact and, uh, I am happy to discuss more about CS with people later. [00:23:53] Thanks, Someone else knows what to do here. ## Discussion — John C. Haltiwanger (00:41:40 – 00:53:43) *Shared discussion for Session 1: Current Employment Statistics; the discussant covers every paper in the session.* [00:41:40] Thank you. Thank you, Ryan. Our discussion is John Halawanganger. [00:42:30] Thank you. So it's it's great to be here. Very nice conference. It's it's a real pleasure to get a chance to talk about these these two papers. These were just two two excellent uh presentations. [00:42:40] Actually these are outstanding papers. I highly recommend reading you say all all the details what what comes across. I won't give given limited time. I won't spend time kind of resummarizing. I think the presentations were great. uh if it wasn't already clear to you from the presentations, it will be when you [00:42:54] read the papers. The expertise underlying these papers is just enormous. The the staff at BLS have been thinking about these issues for for many years have great insights and therb team u essentially took on the ADP and said [00:43:08] we we need to go worry about everything the stats are worried about and and do the best we can. So I'm going to talk more about I'll call it common threads but also sort of complements and substitute kind of issues about when does the ADP data really kick in and [00:43:22] potentially help us when maybe not so much and then and then related to Matthew's opening remarks okay how do we how do we go forward from here not not that I know the answer but but I want to talk about that a little bit so [00:43:35] if if it wasn't clear to you already I mean what's striking about both papers and indeed we we kind of know this anybody's who studied this issue the ben benchmark data is essential. You can't get there from here without really high quality benchmark data. So we we've [00:43:50] known about this for a long time and indeed the the CES is is is a beneficiary of of one of the most timely benchmarks out there. We know for the BA folks in the audience obviously benchmark revisions for GDP uh come uh [00:44:04] at a much slower pace and so and and obviously the ADP paper relied very heavily literally on like every step of the way. Okay, they they know they had a non-representative sample. They had to work worry about weights. They obviously [00:44:17] wanted to do the benchmark revision and then they also had the birth depth problem. Basically the same problem uh that that the CS faces and and and and they they used a similar but perhaps not quite as sophisticated uh method. So so one of the areas I'm going to push on a [00:44:32] little bit is if if given that the benchmarks are so critical, what can we do to improve the benchmarks both in terms of quality and timeliness? I think that's that's a message that emerges uh from from this line of work. So So one [00:44:46] topic I want to talk briefly about and I know the folks in BLS are like experts on this because they've been talking about this for years is why not quarterly benchmarking? Indeed the QCW has the has the potential for uh quarterly benchmarking and this has been studied by BLS staff amongst others and [00:45:00] and I'll talk about what I know about the the limitations why this isn't being done now but we know we know lots of people are using it. uh indeed folks are using the releases of the quarterly QCW all the time to predict the the annual benchmark. So it's it's actively used by [00:45:15] the I'll say uh market participants. It's also being used by BEA in for example in the national company project counts. So it's not as though people aren't using it already. You could say well why isn't BLS doing this? Well I I I as best as I understand it one is [00:45:29] there are known quality differences across the quarters. the first quarter data is of higher quality. But my my my understanding is that's probably a resource issue which is in in in principle if you could devote the same amount of resources that you were able [00:45:43] to devote to the first quarter you could make the second third and fourth quarter as high quality as well and that may be really valuable given what we just heard from these two uh excellent papers. I also heard that and indeed the the BLS paper nicely talks about this [00:45:56] benchmarking is not cheap right to actually do that benchmarking revision takes time and resources. So you could say these are both resource issues and and obviously the folks in this room we are in a resource constrained environment. So how the question is how to get there from here but still given [00:46:09] the importance of this benchmark I think a priority ought to be put on this. So one idea and it kind of came up a little bit in some of the discussion and in the papers we are in a technological revolution going on right now in terms of our ability to process [00:46:22] data in terms of modernizing processing data and AI. The question is can we literally reinvent the systems the old the old SAS and and and other kinds of systems that are that are sitting inside [00:46:34] the agencies and move to Python and AI. Uh is that the place so that we can for example process our administrative data in a in a much more timely fashion. It's also the case and Ryan kind of alluded at this and I'm going to draw broad draw [00:46:48] out a point that he that he raised which is you know amongst other challenges the ADP data has is all the missing industry codes. It's it's non-trivial and the assumption that they had to make was really strong assumption in terms of you [00:47:02] know inside cells missing at random and we and we and we don't think that's probably true at all. It's that's that by the way CS has other missing at random problems as well. So, so you know amongst other things even though uh [00:47:13] clearly the inter entry into ADP is is not um is not true business entry but but there was some hints in the paper that there's some connection between some of the turnover of of entry associated with business entry and and [00:47:27] exit and that's that's a source of non-randomness. [00:47:31] So as Ryan uh suggested at the end the the really the way to get there from here and the question is how do again how do we as a community get there from here if it'd be great you mean you could overcome this aspect of the limitations [00:47:44] of ADP if actually the ADP data could be integrated with the micro uh QCW that's a that's a that's a objective we we should uh we should push towards now the other thing we heard again a common [00:47:57] theme is births and deaths are are hard and they're important right? They play they play a big role in in uh in the nature of the revisions and and they're really hard particularly at turning points uh uh to get right and and and [00:48:11] this is just a reminder maybe was already clear from Brian's very nice presentation. [00:48:16] The the payroll data doesn't help you really at all on this. Okay, you're not you're not getting any help uh uh here. [00:48:21] So the question is can the model be improved? Here's where I think we ought to be thinking about what are really I'll say highfrequency timely indicators that that administrative data that we might be able to tap into to improve our [00:48:36] tracking of births and deaths. So, so one idea I'm a little biased here because I hand in this work at at census the business formation statistics is is basically real- time processing of new new business applications. That flow of [00:48:48] data is weekly almost in real time. we ought to think about whether that data could be potentially used in this context. I've actually also wondered whether whether BLS has ever explored using AI UI claims because that's actually sitting there uh uh and maybe [00:49:02] there's a many papers cited in the in the in the long bibliography that's already been done. I also think and this is related a little bit to the modernization AI point is we know we know that that the tax data both the [00:49:16] payroll tax data and the UI tax data it it's coming in u much more rapidly than than by the time it shows up in the QCW or at census in the business register question is can those systems be modernized so that so [00:49:31] that one is literally we get more timely processing of the registers but but but also early reads so so it may not it may be that there's signal in a in a very early read out of the administrative tax data. Can we can we tap into that? [00:49:45] So here's a place where I think the ADP data really is an advantage. So one thing we know and indeed has been a theme is non uh response rates have been falling and so therefore the all the assumptions that we're making about [00:49:58] non-response bias have become more and more important o over time and I and I think this idea that there's inside cells there's missing at random is just wrong. I think we already know that. [00:50:09] I'll say in work actually that I did with Steve Davis and Jason Faberman and Ian Rucker, we we we had a very a project that matched up the micro uh Jolts data with the BLS LDB inside a project within BLS. We found very [00:50:24] systematic evidence that it wasn't just businesses that as as talked about in the BLS paper that are shutting down that don't that don't uh file reports. [00:50:33] Actually, businesses that are contracting rapidly aren't filing reports. How could we tell? because we actually had the QCW data that we could match up to the to the Joltz data. So I I know BLS is thinking about extending ways of of handling the kind of the non [00:50:48] response bias and and and and I applaud that but but this is a big problem and and and unfortunately getting worse as the response rates have declined here in principle I think that the ADP data [00:51:01] avoids mostly avoids this problem. Although that'd be interesting to hear from that team is that well maybe not completely. Why? because it may be that you don't respond to the BLS survey, but you're still making your payroll is the point. [00:51:12] So, let let me bring this this to a close with a like the really big picture question on the on the table and it's related to Matthew's uh opening remarks. [00:51:23] So, there's a vision that what the stat agencies should particularly be doing in terms of surveys whatever and uh is the benchmark data of administrative and then and then benchmark like surveys. the economic [00:51:37] census, the annual surveys that provide the key benchmarks and that we should be tapping into this naturally occurring transactions data for the high frequency. Could we do that for the PFIs? And I think that's the big question on the on the table and the [00:51:51] question you question we should ask what what do we think about I don't have the answer to that question now but do we think how close are we with the payroll provider data and and the great thing is the ADP uh paper out of the board gives us a a preliminary answer out of this [00:52:04] and what's the preliminary answer we're not ready yet okay that that's not a criticism at all as as Ryan said this is a great supplement to the official data at this point it's it's not a substitute okay we're we're not there at [00:52:19] does that mean we can't get there from here? No. I think actually that's exactly where we ought to be pushing. In some respects, some of Ryan's remarks and so on uh suggest even though ADP is huge given its non-representativeness and that there are other payroll [00:52:32] providers, we we're hear about some of those at this conference that cover some areas like young and small businesses to a greater extent than than does ADP which will help us on some of the dimensions um that they talked about and I talked about. So, so what would I hope [00:52:47] that we as a community do that somehow or another we find the resources to do a large scale demonstration project tapping into multiple uh data providers that ought to be our objective and ideally obviously not only ideally it needs to be with the collaboration in in [00:53:02] this case particularly BLS for this to work and really ideally it needs to be collaboration with BLS in a way that the that this naturally occurring transaction data can be as as we've talked about before is integrated with a [00:53:15] micro QCW. That's the gold standard we ought to be pushing for. The question is how to get there from here. I I know there are there are huge issues about access issues and governance issues and so on for tapping into this kind of data. I I think we ought to partly split [00:53:29] those issues apart. First, we just ought to go see if we could actually tap into this data and integrate it with this fantastic benchmark data, could we could we overcome many of the issues uh that was were nicely talked about here. I'll ## Q&A (00:53:43 – 01:08:20) *Shared floor discussion for Session 1: Current Employment Statistics.* [00:53:43] stop there. Thank you. Um, thanks John. Thanks John. Thanks all [00:54:01] the speakers for being on time. So we have plenty of time for comments and questions. I think I suggest we collect some questions and comments. And David Wilcox, do you want to kick us off? [00:54:20] I had the great blessing of a durable long ve long marriage. My wife reminds me that sometimes it's important to ask the basic questions and state the [00:54:32] obvious things. It feels to me like uh an obvious thing that hasn't been stated is it was implicit in Ryan's comments. [00:54:43] Your discussion was in the past tense. ADP is no longer providing those data to the Federal Reserve Board. [00:54:54] Um so in addition to all the other lessons that we've learned which are tremendously valuable, I think I What I'd like to hear some [00:55:06] recognition of what we take away from that experience. The Federal Reserve Board effort was a live ammunition exercise. We were putting this information in front of the open market committee and allowing them to make [00:55:20] policy decisions because we decided that the product the blended product was high enough quality to warrant that and suddenly it was cut off. What happens if [00:55:33] What happens if >> that that was [00:55:50] >> okay? >> Thanks. [00:55:53] >> Could you could you say who could you announce your name for? [00:55:55] >> Sure. Sure. My name is Michael Steer, professor assistant professor at the University of Toronto, collaborator of the many of these uh um administrative uh private sector data projects. I want [00:56:09] to just raise room. There's another angle here, right? We talked about administrative private sector data. The other angle is improving the administrative data that the government gets by requirement. And uh my former [00:56:23] collaborator Peter Ganon is hosting a conference in November about the UK's experience where the government reformed payroll reporting requirements to require companies to report more regularly [00:56:36] uh to the government to their payroll. So the idea would be the analogy here would be rather than making do with the best we can of the QCW to benchmark all [00:56:48] these private sector data what about um improving the QCW of course that has its own challenges and uh changing government policy but I think it's worth thinking as we think about modernizing [00:57:03] our employment measures about both things because right now we're doing what we can with what we have. There may be other angles and other countries that [00:57:12] we can look to for um other models here. >> Thanks. [00:57:33] >> Hello, I'm Hijuan from the Federal Reserve Board. So I have a question about the you know non-random missing observation and that potential bias to the seniors and the joy. So as John mentioned and also you know mentioned by [00:57:46] Ryan um that certainly this missing observations are not responses in the you know established survey um is not random. So there's evidence for it and certainly um it can be a you know challenge you know the payroll data in a [00:58:01] sense that that actually means the ADP data can be actually more um timely in detecting recessions or expansion especially there is a you know pattern of bias that you know like underperformers tend to not respond to [00:58:15] the surveys you know so um but I was wondering you know but additional information to bear is always help us to identify the patterns of bias arising from nonresponses so I was wondering whether you have a chance to study this [00:58:29] or you know think to think about the strategy of you know about this bias at is the direction of magnitude based on the AD. [00:58:39] >> Thanks, John. Stable houses. >> Yeah. Yeah, it's on, John. I think it's [00:58:54] on. >> Uh, so Brian, show us some examples of periods in history where FRBA did a jobs suggesting CDS be pulled back in that direction. I guess the question is are [00:59:09] there other periods in history where it went the other direction and if so what would we say about different types of uh I guess points of business okay I will use uh u organizers [00:59:34] privilege so I I I think David's question and and Michael's uh Michael's are actually related. So if you if you continue to have alternative data feeds [00:59:47] as ad hoc measures, this kind of thing can happen and and uh these are basically dependent on contractual relationships. other hand if you if you take the data not alternative data but it's the data and start feeding it in to [01:00:01] the official process some of it could go off to the states for benefits processing which is uh some of it could go to the BLS for uh statistical processing those could be coordinated the data become for the payroll [01:00:15] providers or the individual payrolls of firms in in this modern way but it would the process would be integrated with the data collection And that that's a different infrastructure than sort of ad hoc relationships of purchasing or [01:00:29] cajoling data providers. So we we have a whole panel on this tomorrow. So I hope we take this we can think about this and discuss this further. Did the if there are not further questions like I did the uh presenters want to have a chance to [01:00:43] say something? Uh >> uh is it >> lean? Could you introduce yourself just [01:00:58] for the record? >> Yes. Just just say your name for the just >> Yes. Thank you. [01:01:05] >> No. So John's questions for the AP side. It's like non response is not a problem. [01:01:16] universe once you're in. [01:01:23] We always had questions whether it switch yourself anymore. So it's something we kind of looked at trying to find patterns the [01:01:36] same way. Okay. Okay. Okay. [01:01:45] And then um to John's question about revisions. So again, I think we don't have much of a problem [01:01:56] with first and third as we get monthly semers in, but we don't have a third significant third month happening the same way CBS does [01:02:10] because folks just don't get the reports on time. So for those kinds of third revisions, ADP was probably has significantly smaller revisions. That said, we saw the benchmark figures and [01:02:23] there were times when um had benchmarks did so we don't like cherry picking um particular cases too much but that's [01:02:37] >> I was interested about direction and size. [01:02:44] like are you pulling back towards CVS or uh are you suggesting that the C are you pushing against CDS is going to get revised? [01:02:55] >> Um I mean we might need to look at the data some more but again we're not you know ADP wasn't revising much so CBS will sometimes be revising occasionally they would be revising away from ADP but [01:03:08] >> I mean there were years when uh ADPFRB benchmark revision was uh bigger I mean many years in which it was bigger than CVS and there were years where it was bigger than CDS and in the same direction. So it was this so if you had [01:03:21] taken too much signal in that year it would have aired. I mean conveniently in these really big CS benchmark revisions that had done really well. So when the states were high it was helpful but there were other times in 2017 2018 it was on that chart I showed when it [01:03:35] wasn't so good. So, uh, you know, macro forecasters, which all of our great labor forecasters are actually here right now, have to decide how to weigh these different signals and and what to do about it. I'll make one comment on uh [01:03:49] this point about the holdup problem. So, this I'm just going to plug a statistic we produce at the Fed called the Industrial Production and Capacity utilization statistic. And this is a statistic that for has been going for over a hundred years, but for that whole time it has been a blended data product. [01:04:03] And so we have lots and lots of of private sector sources from all over the place uh that feed into that. And over many years we have lost many many uh sources and I think that the cost of [01:04:17] losing a source it's partly dependent on obviously how important that source is. [01:04:21] If you were using something like ADP which is so huge uh and if you were very dependent on it it would be very costly to lose it. But at the same time I think the cost of losing these sources depends a little bit on how good your benchmarks are. how soon can you catch it and can you replace it? Can you come up with [01:04:35] some sort of a workaround? And um I think the BEA has lots of experience with this in the Nippas. They've had lots and lots of sources over the years, but I think you've come gone. And so it's a very tough problem. Uh and how you design a system, you know, you want to keep it in mind, but I think it's [01:04:49] also a surmountable problem. Katherine, >> I just had a a slightly different thing to ask about, which is a lot of times [01:05:01] when people talk about augmenting the official statistics with other data, they're interested in being able to produce more disagregated statistics. And when I, if I'm [01:05:16] remembering your paper correctly, you said you just tried to wake things up at the national level. you didn't try to reweight the geographic bubble um and and it didn't look like you were interested in producing geographically [01:05:30] disagregated statistics, but I'm wondering what thoughts you had about how incorporating AP data in conjunction with the CES data might help with that [01:05:40] or whether it wouldn't be that helpful. >> Yeah. So uh yes uh we we didn't do waiting by region. We did over the years many times look at sort of regional patterns uh like if there was a hurricane or something. So we did [01:05:54] sometimes do that. Um one challenge in probably a lot of these data sets is that sometimes you'd like to imagine if you know what kind of business unit you're dealing with. Are you dealing with an estate or a firm? But as we wrote in the paper, it wasn't always so [01:06:07] clear. We treated these businesses as establishments. Some of them might have been firms. And if you have uh firms masquerading as establishments and that go try to do regional stuff, you're going to be misled. And so a very large multi-unit firm, if it's looking like [01:06:21] it's all located in one place, it can really throw things off. So, uh the data was certainly that sort of thing. Um but it again comes down to these tricky like paying very close attention to [01:06:34] measurement choices. I'll just comment real quick on the non-response issue and data that are not missing at random and when the respond is related to what we're trying to [01:06:48] measure that really problematic and so what we're trying to do is look at are there other things that you could observe that we could make some adjustments for so uh I know John mentioned in relation to birth depth which we actually did look at back in [01:07:02] 2013 and uh I think it worked okay Um this last round we really tried to tie it to CS CS data but um we've been looking at um this is this is really [01:07:15] messy but um one looking at count's data linked like this but um if a business that's not responding lighting [01:07:29] up a lot of people that probably tells us something about what's happening there. So looking at stuff like that we're able to some of these Friday data um with CS survey data we might be able to make that as well as [01:07:44] we want to do I'm sure about work but some of them do >> uh thanks so I want to thank the the uh [01:07:57] authors for these papers these papers were commissioned for this uh for this conference and it got us off to a great start by setting out the issues and showing showing how much progress has been made but and I hope it sets up the [01:08:11] the foundation for further progress. We'll take a 20-minut break and return promptly at 2:40. Thank you.