The structure of the Evidence Act shows the importance that Congress placed on inference in the process of generating evidence. The first Title set up the organizational infrastructure – the people who would do the work, like Chief Data Officers, Chief Evaluation Officers, and Statistical Officials. The second Title established the importance of data access. Title III highlighted the importance of statistical agencies in producing evidence. Data literacy, in other words, is not just a matter of operations and coding. Statistical thinking is required, and that means an understanding of the data generating process and how data came to be. Errors can show up in many ways. Selection bias is particularly prevalent in the data generating process in education to workforce transitions. Clearly students (and their parents) will take possible future earnings into consideration when they choose a major. As a result, we only observe earnings data on those who have self selected into the major in which they think they are going to get the highest earnings. One reason for Jim Heckman’s Nobel Prize in 2000 was the development of a technical approach correcting for selection bias – which was particularly helpful in understanding why historical analysis of the link between women’s education and earnings outcomes might be biased due to women’s higher reservation wage. Another common challenge is in data processing prior to analysis. In the Census Bureau’s Post Secondary Employment Outcomes data series, for example, outcome measures are based on the calendar year post graduation (which will commonly blend two academic years) and the earnings measures impose two labor market attachment restrictions (an earnings threshold and a number of quarters worked) that are likely to bias estimates upward for marginally attached workers. It pays to be very careful not just with your own decisions, but to examine decisions that others have made.

Inferring future outcomes based on past data carries an inherent risk: the assumption that history will repeat itself. While the past provides a plethora of insights, it doesn't always account for novel conditions or unforeseen disruptions. For instance, socio-economic landscapes, technological advancements, and geopolitical shifts can drastically change the paradigms we operate in. A model developed on data from a stable economy might not hold during an economic downturn. Similarly, labor market dynamics of the past might not necessarily reflect the evolving nature of professions and industries. Relying solely on historical data to predict future trends can therefore lead to misguided policies or decisions that don't cater to the evolving nuances of the real world.

The Google Flu Trends series utilized Google searches related to flu symptoms to estimate flu activity in various countries. While the Google predictions were initially accurate compared to CDC data for 2009-2011, it predicted levels nearly twice those of the CDC estimates for the 2012–2013 flu seasons.(1) This discrepancy, as Lazer et al. (2015) highlight, stems from big data hubris—the assumption that data volume compensates for its inadequacies—and algorithm dynamics, which involve the evolving nature of algorithms. Notably, Google's changes caused flu-related searches to artificially surge due to shifts in its search suggestions.

Such algorithmic changes are by no means unique to Google, as Twitter and Facebook also modify their platforms to improve user experience. However, such changes, because they change the underlying data generating process, affect the utility of their data for analysis. Data that provide reliable estimates today may not do so tomorrow, as data collection methodologies evolve. The labor market data produced by proprietary companies like Lightcast must be equally carefully monitored to understand how the data were generated. Understanding the limitations of their proprietary algorithms – such as for The Google Flu Trends case above – can be challenging because the accuracy of the algorithms relative to specific benchmarks are not publicly measured. For example, is an increase in job postings due to an increase in the number of companies posting to a Lightcast site, or an actual decline in the number of postings? Or is it due to an uptick in an industry that is over-represented in Lightcast sources? How is seasonal adjustment handled? Are careful benchmarks and documentation provided at the same quality level as that produced by statistical agencies? Another example of private sector sources of data is ADP data, which produces employment data every month before the BLS employment numbers. The ADP coverage and methodology is different, the differences are documented both through a careful technical report and through its partnership with academic organizations including the Stanford Digital Economy Lab (2),

The FCSM data quality framework introduced in week 3 (Figure 1) provides a helpful rubric to assess whether new types of data are representative of some known population (and deduplicated), mapped to commonly accepted standards, and is open for scientific replication and review. For more on this topic, see Chapter 10 in the textbook (3). Figure 1: The FCSM data quality framework. Source: FCSM

The rapidly changing data environment and changing data needs means that it is unwise to rely on static models. Models should be recalibrated as new data become available, as, for example state data sharing efforts expand through regional state data collaboratives. Those collaboratives can also be used to generate a common measurement system and interoperability blueprint that connects data across programs and aggregates the results to generate statewide, regional, and even national labor market trends. The required outcome measures – such as job quality – will change with legislative initiatives, such as the CHIPS and Science Act, the Inflation Reduction Act, and the Infrastructure Investment and Jobs Act–which will fundamentally change the economy.

It’s important also to remember that developing models and analysis does not have to occur in isolation. Analysts can leverage the data science expertise of local universities to build their capacity to use their own data and keep informed about new techniques that can be used to inform and guide decisions. States are increasingly focused on sharing knowledge and developing scalable products and services– ranging from a skills-based talent marketplace in Alabama, to integrated service delivery systems in Arkansas, to unemployment-to-employment portals in Illinois and the focus of this training - multi-state education to workforce dashboards in Kentucky, Ohio, Tennessee, Indiana, New Jersey and Virginia - with other projects underway in states like Texas. The National Governors’ Association which is leading a skills-driven states’ community of practice that supports state leaders that are developing infrastructure to link digital learning and employment records.

References:

Lazer, David, and Ryan Kennedy. "What we can learn from the epic failure of Google Flu Trends." Conde Nast10 (2015): 67-85.
ADP National Employment Report. 2023.https://adpemploymentreport.com/ ; see The Fed Cares a lot about jobs data - but it may be getting mixed signals. CNN Business. https://www.cnn.com/2023/09/04/economy/labor-market-bls-adp/index.html for a non-technical description
Textbook: https://textbook.coleridgeinitiative.org/chap-errors.html#sec:10-1 and https://textbook.coleridgeinitiative.org/chap-errors.html#sec:10-3