One of the most difficult challenges of dealing with administrative data is that you are often dealing with a population that is not the full population – it is the population for which data are available. So in our education-workforce case, information is not available on students enrolled in private sector educational institutions. Information on jobs is only available for students who get jobs within their state – although states are banding together to share information across state lines (so far, Indiana, Tennessee, Ohio and Kentucky are participating in the activity, with other states joining shortly). That means that analysts have to be very careful in cautioning policy makers about the limitations of the analysis. It also means that, missing data on jobs, for example, may be a consequence of a graduate not being able to get a job at all or having got a job out of state. If the former is the case, and missing data are dropped from the analysis, estimates of the impact of education on job quality will be biased up. If the latter is the case, and missing data are dropped from the analysis, then estimates of the impact of education on job quality will be biased down. Worse, if there are different propensities to move out of state for some institutions – like Northern Kentucky University, which borders Cincinnati – the biases will be different by institution. The same holds true if there are different propensities for different majors or demographic groups. The results of biased analysis will be biased evidence and possibly unintended consequences for policy.

Start with a cohort. For analytical purposes, it is often helpful to identify a specific cohort to trace outcomes over time because that reduces many confounding factors. The choice of cohort will depend on the theory of change. If, for example, the theory of change includes a possible financial intervention, because the hypothesis is that students are not completing due to financial pressures, the cohort would start with individuals enrolling in a given academic year. If the hypothesis was that completers were leaving the state because of better jobs elsewhere, the cohort would start with individuals who completed their degree in a given academic year.

State results relative to the specific population used in the data. This practice ensures clarity in communication and prevents misinterpretation. It is crucial to explicitly recognize that the analysis --results, and insights derived—may be relevant only for that subset and not generalizable to the broader population. It is also important to be very clear about the unit of analysis – a good example is in the technical documentation in the MSPSR report, which notes that the Kentucky data “encompasses all 4-Year Private (Association of Independent Kentucky Colleges and Universities [AIKCU]), 2-Year Public (KCTCS), 4-Year Public Comprehensive, and 4-Year Public Research institutions. Proprietary institutions are not included in this report due to a lack of data.” (see Figure 1)

Figure 1: Source: Selected
from the Triage project

Use alternative data sources to get a sense of the degree of any bias. For example, missing data could occur because completers didn’t get jobs, or because they left the state. The LEHD Job to Job Flows Explorer can give a sense of the flows from one state to the next. But be aware of the quality of data sources. The Job to Job source documentation is available here. While the BLS Local Area Unemployment Statistics are often used, they have had challenges in recent years (here and last page here )

Figure 2: Source (https://j2jexplorer.ces.census.gov/)