Good inference requires that the analyst think about the data generating process not just in aggregate. Screenshot 2023-10-12 at 08.53.48.png A major challenge for inference, in other words, is the degree to which missingness can bias the results. For example, look at the earnings and employment outcomes for graduates from Northern Kentucky University in the MSPSR screenshot in Figure 1. About half of the graduates get jobs in Kentucky (the grey portion of each bar); about half in Ohio (the blue part of each bar). The proportion of education majors getting jobs in Ohio is much higher than in other majors. The geography, in this case, is likely due to the fact that Northern Kentucky borders on Cincinnati, Ohio. Many Northern Kentucky University students are likely to have found jobs in Ohio, and may no longer show up in the Kentucky wage records. Both the employment and earnings outcomes are likely to be biased downward as a result. While it is possible to try to impute the missing values, a much more proactive approach is to build data infrastructures that enable states to share data across state lines, which is what is being done by the multi-state data collaboratives.

Figure 1: MPSPR report of earnings and employment outcomes
for students graduating from Northern Kentucky University

Another example of understanding the data generating process the change in survey methodology that occurred as a result of COVID. Because the fear of infection meant that interviews needed to be made over the phone, rather than in the field, it was harder to reach respondents, much less get them to answer their phones. In the case of the Current Population Survey (CPS), a key survey for labor market data, the month-over-month nonresponse rate increased by 58 percent, and the share of Blacks and of White Hispanics dropped substantially. (1) The impact on local labor market data was substantial. In the case of one state, Michigan, the non-response rate – the level of missingness – was particularly high in Detroit in every month from January to September in 2021, and was not fully corrected for by the federal government. Once the adjustment for non response error was identified and corrected, the Michigan October unemployment rate had to be revised from 4.6 percent to 6.3 percent. What was not so obvious was the repercussions on the unemployment rate in other states. Because the CPS sample size is too small to produce valid unemployment numbers for individual states, the unemployment numbers are estimated for an entire region – in this case, BLS region 5 - and then proportionally imputed to all states in the region. BLS region 5 includes other major states – Illinois, Indiana, Ohio, and Wisconsin – and the failure to adjust Michigan's data for missingness had a knock-on effect on those other states. Thus, for example, the Illinois governor was making decisions based on wrong data for Illinois - not because Illinois data were incorrectly adjusted, but because of a federal adjustment error for Michigan data. In particular, Illinois’ unemployment was much lower than was reported for each month from January to September of 2021. As the November 18, 2021 Illinois press release pointed out, the errors were identified when “Illinois and another East North Central Division state raised concerns about their monthly 2021 statewide labor force estimates”.

Administrative data have very specific challenges, because changes in the rules for administering government programs can result in the arbitrary inclusion or exclusion of particular groups. In the case of unemployment claim data, workers in the “gig” economy were historically not included (2). But the rules were changed with the COVID pandemic to cover “gig” workers, and so part of the reason for the surge in unemployment claims was that coverage increased as a result of the Pandemic Unemployment Assistance Act. After COVID, "gig" workers are no longer included, and so analysts need to remember to adjust their analytical framework.

Inference also means examining the impact of changes in data collection for sub-populations. Screenshot 2023-10-12 at 08.56.28.png This can be particularly since they have, by definition, smaller sample sizes. Thus, even when data are available, inference is likely to be less precise. For marginalized or even smaller sub-populations, missing data and inadequate sample size can silence the data’s ability to address these groups’ unique experiences. In the MSPSR report (Figure 2), for example, data are much less available for Blacks with diplomas. It is essential to recognize these data blind spots, and when possible, rectify them so that no group is left behind. In the next section, we will talk more about ways to address the resulting ethical dilemmas in evidence-based decision-making.

Figure 2:Earnings by Demographic Group (MSPSR)

Reference:

1. Jason M. Ward, Kathryn Anne Edwards, CPS Nonresponse During the COVID-19 Pandemic: Explanations, Extent, and Effects,Labour Economics, Volume 72, 2021,

2, Commissioner's Corner Why This Counts: Measuring "Gig" Work, March 3, 2016.