Our post about the struggles with surveying nonvoters and young voters focused on the Cooperative Election Study (CES). This week, the headlines focus on the Current Employment Statistics (also CES). I have not worked with this CES data, whose importance is highlighted in very beginning of the textbook by Groves et al.:
At 8:30AM on the day before the first Friday of each month, a group of economists and statisticians enter a soundproof and windowless room …
Those in the room are professional staff of the U.S. Bureau of Labor Statistics (BLS)…
A household survey [CPS] produces the unemployment rate; an employer survey [CES], the jobs count…
However, only when decision makers believe the numbers do they gain value. This is a book about the process of generating such numbers through statistical surveys and how survey design can affect the quality of survey statistics.
On Friday (August 1, 2025), the Bureau of Labor Statistics (BLS) reported revisions to the jobs count from the CES:
Revisions for May and June were larger than normal. The change in total nonfarm payroll employment for May was revised down by 125,000, from +144,000 to +19,000, and the change for June was revised down by 133,000, from +147,000 to +14,000. With these revisions, employment in May and June combined is 258,000 lower than previously reported. (Monthly revisions result from additional reports received from businesses and government agencies since the last published estimates and from the recalculation of seasonal factors.)
A search for “nonresponse Current Employment Statistics” yields this BLS paper from 2003, when the CES moved from a quota sample to a probability sample. The focus is nonresponse, which includes both nonreporting and late reporting. They write:
assume nonresponse is ignorable within defined estimation cells…estimation cell (defined by industry and, for selected industries, region)
One useful path for further research would appear to be refinement of estimation cell definitions, in an attempt to define a more appropriate ignorable nonresponse model… Factors for consideration beyond industry, based upon research to date, would include size of establishment and geography. These factors are utilized in the sample design, but not (with a few exceptions) in the definition of estimation cells…
A second path would be to explore other ignorable nonresponse models…
This mirrors our post from last week, where we discussed both approaches.
The current BLS Handbook of Methods: Calculation says:
Cells are defined primarily by industry. Geographic stratification is also used for some construction and government industries.
The current BLS Handbook of Methods: Design says:
The sample strata, or subpopulations, are defined by state, industry, and employment size, yielding a state-based design. Sampling rates for each stratum are determined through a method known as optimum allocation
Beyond skimming these helpful documents, I have not yet learned the details of their methods. My naive understanding is that establishment size is used in the stratification and not the estimation. My naive question is: why ? I am sure the folks at BLS have thought about this carefully and I am curious.
p.s. Andrew tells me his mom used to work for the BLS, maybe he can share some stories ?
The account for the revisions that I heard (I’d like a link to the precise changes so I can verify this myself) was that at least half of the revisions were due to state and local government employment. And that data was submitted late – driven by staffing challenges and mostly related to school staffing. This is a different sort of non-response, clearly not random and also related to uncertainties driven by current administrative policies. If this is accurate, then I wouldn’t characterize these as general features of the BLS sampling, but more as unique challenges posed by current policies. As this is a controversial and current political issue, it would be good to see the links to exactly what changes occurred and why, and how these relate to traditional sampling issues with BLS surveys. Can anybody provide direct links?
State and local hiring was about half or so of the revisions to both May and June. But the May revision was associated with updated seasonal adjustments.
Thanks, Dale ! Fully agree on providing all the links.
In my post, I linked to and provided the summary from the August 1, 2025 Employment Situation Summary:
Let’s break this down by industry.
The +144,000 comes from Summary table B. Establishment data here. Government is at 7,000.
The revised +19,000 comes from Summary table B. Establishment data here. Government is at -50,000.
So the revision to Government numbers was 57,000, which is almost half of the total revision of 125,000. This lines up with what you heard, Dale, right ?
So the full BLS survey methodology is NOT transparent to outsiders. Why ?
The latest absurdly gross “Revisions” to the BLS May/Jun ‘data’ and the long BLS history of big revisions — demonstrates that such BLS ‘reporting’ is fundamentally unreliable.
Yeah! It’s a house of cards! Let’s tear it down and replace it with a politicized bureau! That will certainly give us neutral, reliable data! (\sarcasm, for the record)
“the long BLS history of big revisions — demonstrates that such BLS ‘reporting’ is fundamentally unreliable.”
I would go through this fragment and describe in detail the multiple layers of absurdity present. But I have a feeling it’s a trolling post intended in bad faith.
Dorset, I think it is a sign of transparency and reliability to revise reports when more data comes in.
If you email BLS they are very responsive and can explain the details. Employment enters the estimation because they are taking the employment weighted average of unit growth rates within each size-industry-state cell. Not sure if that is what you had in mind.
Split sample, it’s very cool to hear that BLS is so responsive, I’ve heard that as well ! I have just emailed them and will report back !
You say “within each size-industry-state cell”, but the current BLS Handbook of Methods: Calculation that I link to in the post says “Cells are defined primarily by industry. Geographic stratification is also used for some construction and government industries.” They don’t mention size there.
One of the things we lost when 538 was shuttered is that they had a nice dashboard for the job report with some historical numbers and the revisions right there. But I did come across this article from 2020 when the Trump administration had the opposite problem: an unexpectedly good report. https://fivethirtyeight.com/features/a-good-jobs-report-does-not-mean-a-rigged-jobs-report/ The TLDR is that the report would be very hard to rig and lots of people would quickly figure out if it were.
Thanks, Alex ! That article is super helpful:
Imagine an organization where the leadership is chosen based on being the best liar. Then imagine believing the stats generated by that organization.
I assume you are referring to the impending changes at BLS rather than its historical performance. Then imagine an Executive Branch where members are chosen based on being the best liars (with a provided script), and then imagine the “information” generated by that administration. Doesn’t take much imagination, though.
More like this scene: https://youtube.com/watch?v=YvDcJydVcHo
A couple of times over the past few decades I needed clarification about the methods used for particular BLS numbers. So I called them, got connected to the person responsible for that piece, and all my questions were answered. Their personnel were absolutely professional and generous with their time. There are a lot of very specific issues that go into initial estimations and subsequent revisions, since their product is so granular. The headline numbers don’t capture this.
Thanks, Peter ! So many folks have been saying this, the BLS is impressively transparent and helpful. I have just emailed them and will report back !
I’ll put this here, although it doesn’t quite fit and there might be other threads where it is more appropriate. But I think it is important: https://www.whitehouse.gov/presidential-actions/2025/08/improving-oversight-of-federal-grantmaking/. For all the problems with research – which the executive order pretty much gets right – the solution is surely worse than the problem. Politicizing all research funded by the government, and implemented by our Supreme Administration, is not what I see as improving the problems with poor research practice. If you read the actual text of the order, it is hard to imagine a worse intrusion on research or one more likely to make things worse.
Agreed. It’s almost as if it’s designed to have the most pernicious effect.
I’m not sure that the executive order gets the problems with research pretty much right. Its first section (“Purpose”) describes a sort of pastiche of the stuff that’s thrown around the electronic infosphere to discredit science. So, for example it’s asserted: ”In 2024, one study claimed that more than one-quarter of new National Science Foundation (NSF) grants went to diversity, equity, and inclusion and other far-left initiatives.”. One might have thought in a government report about a topic on which they have direct oversight, that this should be known one way or another and wouldn’t have to rely on circuitous and self-absolving language like “one study claimed…”.
IMO the “problems with research” are overstated especially wrt to the Biomed/Chem/Physics areas although one might reply that researchers and research structures have brought this imposed oversight upon themselves by not responding as well as they could have to justified criticism. From my european perspective US researchers are between a rock and a hard place since even efforts to address some of the issues around reproducibility, data manipulation and so on just become weaponized in support of dreary agendas. Elizabeth Bik who many people will know of as a “datasleuth” who hunts for evidence of (especially) image manipulation in scientific papers has pointed out this problem in a recent news article in Nature (9th July “Research-integrity sleuths say their work is being ‘twisted’ to undermine science”):
Thanks, Dale, Joshua, and Chris for these thoughts. I do think this discussion belongs here.
I was hesitating to ask methods questions because I was worried it would sound like I’m questioning the integrity of the BLS researchers (I definitely am not).
1) in this case, I am quite sure they are doing something reasonable and I would like to understand it (I did email them !).
2) in other cases it is good for science to question whether a method is reasonable.
3) in very different cases it is good for science to question the work’s integrity.
The other issue is that errors are correlated across time and space suggesting we could be using more data…
Pretty clear it is not even close to the best estimate you can provide
Thanks, gaurav ! I assume methods can always be improved. This is why I’m happy the BLS is so transparent and open to discussing their methods. I will keep you posted with what they say !
It’s good to see Bik identify that problem. She’s clearly a heavyweight and it means a lot coming from her
Unfortunately, Kennedy, Battacharya, Makary, Prasad et al. have certainly weaponized valid criticism. They also claim to be doing this under the banner of de-politicizing science. It’s those kinds of claims that I jokingly say makes me think the tech bros might be right, and the only explanation for this world is that we’re living in a simulation.
Chris – that’s behind a paywall but I see the author isn’t Bik. Is she quoted in the paper?
Yes, it’s a news article written by a Nature science reporter. She interviewed several of “datasleuths” who are quoted in the article along with Elizabeth Bik. I don’t think we’re surprised by their comments but it’s useful to see them stated.