“One third of a data model usually consists of common constructs that are applicable to most organizations, on third of a data model is usually industry specific, and one-third of the model is specific to the organization” (1)

The potential to do new and exciting empirical analysis is constantly growing, as a result of both new sources and of new types of data(2, 3). Data are no longer just carefully produced by statistical agencies, but are derived from a host of other sources, such as social media, credit card transactions, job listings, cameras and cell phones to cite just a few. Unfortunately, the potential to do really bad analysis is growing just as fast. It is natural to want to start analyzing data right away, but the GIGO (Garbage In, Garbage Out) principle is as important in data as it is in any other work. Getting data right means understanding the business processes that generated the data and constructing a data model that enables you to map those business processes in a stylized and repeatable way.

The focus of this class is learning how to use administrative data. This type of data is a by-product of administrative processes when government programs are implemented(2). The significance of administrative data has surged and the use of such data is increasingly required for government policy since the passage of the Evidence Act (3, 4). Most policy analysts have been trained to use data produced by statistical agencies, but there are massive differences between statistical data and administrative(5).

With statistical data, the data has been generated by an army of survey statisticians, survey methodologists, and field interview staff. They have carefully developed survey questions that capture the concepts of interest, identified the population, developed a representative sample, collected the data, processed it, imputed missing values, documented it, applied disclosure protection, and made it available for use. The measures are predefined.

With administrative data, the data has been generated by business processes, with little documentation, and the analyst has to generate the measures. Understanding how to think about administrative data and how to do high quality analysis really means thinking about how the data was generated, and what questions it can be used to answer. Every administrative data project should begin by mapping high-level questions to a set of measurable concepts that can be used for analysis. The challenge is to understand the business process whereby data are generated, for what purpose, and how closely the concepts can be mapped to reality – the data model.

Take, for example the measurement of the common (and very important) concept “income”, and how the construction of a measurement of income differs as a function of which of the two sources is used.

Statistical: Income in the Current Population Survey is derived from 18 different questions asked with a carefully designed survey of a carefully designed sample of the population. The information is collected using both personal and phone interviews of selected individual in about 60,000 households across the country every month. Missing values are imputed. Sample weights are reported (see the Current Population Survey reading).
Administrative: Total wages vary by state but typically reflect payments made to an employee by a covered employer (Unemployment Insurance (UI) wage record report) The monthly information is collected every quarter from every UI Covered employer about every employee in the firm, and reported to the state Department of Labor for the administration of the Unemployment Insurance Program. Missing values are not imputed. No sampling weights are applied, because the data are collected from the population of UI covered workers (See the UI wage record reading).

As the opening quote highlights, one third of any data model will be standard, one third will be industry specific, and one third will be specific to the organization. The standard entities of interest in policy analysis are usually organizations (in this case, schools and employers) and households/family units (students) or individuals. The industry specific part of the model - because the use case is tracing education to workforce outcomes - is enrollments and completions, employment and earnings. The organization specific part in this case is the state of Kentucky; each state has different educational and workforce systems.

In the case of criminal justice policy, the organizations could be prisons and employers, while individuals could be the formerly incarcerated population. In health policy, the organizations could be hospitals, doctors’ offices, emergency care services etc., and individuals could be patients.

References:

1. Silverston L, Agnew P. The data model resource book: universal patterns for data modeling Links to an external site.: John Wiley & Sons; 2009.

2. Cunningham J, Hui A, Lane J, Putnam G. A Value-Driven Approach to Building Data Infrastructures: The example of the MidWest Collaborative. Harvard Data Science Review. 2021.

3. Advisory Committee on Data for Evidence Building. Advisory Committee on Data for Evidence Building: Year 2 Report Washington DC 2022.

4. Kentucky Statistics. 2023 Multi State PostSecondary Report Technical Notes. 2023.

5. Hand DJ, Babb P, Zhang L-C, Allin P, Wallgren A, Wallgren B, et al. Statistical challenges of administrative and transaction data. Journal of the Royal Statistical Society Series A (Statistics in Society). 2018;181(3):555-605.