Measurement is hard in general, but it’s very hard in public service. Simply put, what you measure is what you get. It’s ironic that one of the early examples of “dysfunctional measurement in an organizational setting” (written up in page 6 of (1) is an employment office in the 1950’s. The (unnamed) government office had the stated purpose to serve workers seeking employment and employers seeking workers. Early in the study period, employment interviewers were evaluated by the number of interviews they conducted, so of course, the interviewers maximized the number of interviews and therefore didn’t spend too much time locating jobs for clients. Then, when the measure changed to measuring the ratio of placements to interviews, the interviewers destroyed the records of interviews page(1). In our use case, a focus on job quality that measured only job duration would encourage institutions to get their graduates to stay in jobs they were not suited for; a focus on job quality that measured getting jobs immediately would not set them up for the best job possible; a focus that measured high earnings alone might not set them up for jobs with earnings growth…and so on.

In the realm of public service measurement, an analyst must be acutely aware of the incentive structures that specific measurements can create. Every metric selected inevitably prompts certain behaviors, and it's vital to predict and understand those behavioral shifts. Metrics should not only reflect the desired outcomes but should also be chosen with an eye toward the potential for unintended consequences. An analyst should always ask: “How might this metric, if it became the primary focus, distort behavior, and divert us from our actual goals?”

Take education as an example. If a school system primarily emphasizes and rewards high standardized test scores, educators might be inadvertently pushed towards "teaching to the test" rather than fostering a holistic and comprehensive educational experience for students. In such an environment, crucial aspects like critical thinking, creativity, and practical skills might be sidelined. This bias could result in students being well-prepared for standardized exams but potentially lacking in other equally vital life and career skills. Thus, an analyst examining educational outcomes must recognize these potential pitfalls and ensure that metrics provide a balanced view that prevents such distortions. A general discussion around the effects of measurements in performance metrics can be found here (2).

Measurements also matter in the way they shift emphasis to short-term or long-term outcomes. For example, focusing only on immediate employment rates of graduates may overlook other important factors like long-term career growth; i.e., pushing students into quickly finding jobs to boost employment rates might compromise their long-term career development. Over time priorities might shift so that measures need to be updated to reflect societal and economic changes; e.g., traditional measures focused on full-time employment may not capture the reality of an increasingly gig-based job market.

References:

1. Austin RD. Measuring and managing performance in organizations. Addison-Wesley; 2013.

2. Fisher, N. (2021). Performance Measurement: Issues, Approaches, and Opportunities. Harvard Data Science Review, 3(4). https://doi.org/10.1162/99608f92.c28d2a68