One of the most important activities for data scientists – and a major reason that they are paid so much – is linking data records. It has many terms in computer science, including “entity or reference resolution”, “disambiguation”, “duplicate detection” and “object consolidation”. The simple reason for its importance is that the better a company can connect a variety of different datasets, the more profitable they tend to be – about 6% more profitable according to some estimates (1). It enables companies to be more efficient in targeting information, resulting in increased sales and lowered costs. That’s a big reason companies ask for so much personal information – our birthdate or our email address – when we visit a website, buy something at a store, or use a social media app. Those types of link keys make it easier to connect individuals across different datasets. It’s so important to companies, that, for example, in 2022 when Apple changed its privacy settings for app tracking to be opt-in rather than opt-out, Facebook reported that the 2022 cost in lost sales would be about $10 billion.

The reason that the linkage of different datasets is so difficult is that unlike surveys, where you can ask the same person questions about their education, work, and neighborhood, when data come from different sources – information about education is in a school transcript, Screenshot 2023-07-31 at 10.42.15.png

information about work and income is in payroll data, and information about the neighborhood is in google maps. Thus, it’s a lot of work to ensure that the data match to the same person. The terms “entity resolution” or “deduplication” are common technical terms in computer science for data linkage, because they refer to the precision with which computers can decide that Person (or business) A in File X and Person (or business) B in File Z are the same person. For a great overview, read the very clear introduction by Binette and Steorts(2)

In a business world, more precision is worth a lot of money. The reason that firms like google offer “free” services like Google Maps (which relies on linking your location to Google Earth, traffic data, and your input) and grocery stores offer “discounts” is that they are then able to link many sources of data together to get more insights about you. The more information you provide, the easier it is to ensure that the links are accurate. As the saying goes, if you’re not paying for the product, you are the product. How much is the additional information worth? It’s hard to know exactly, but one study showed that when Europe, worried about the privacy impact, implemented the General Data Protection Regulation (GDPR) which limited data linkage, the most affected firms saw a drop in profits of 8% and in sale of 2%(3). Jeff Jonas, who was Chief Scientist at IBM, started a company providing entity resolution services and the value to companies can be inferred from the price list in Figure 1.(4).

Figure 1: Pricing list for entity resolution for Senzing.ai (retrieved July 22, 2023)

References:

1. McAfee, A., Brynjolfsson, E., Davenport, T. H., Patil, D. J., & Barton, D. (2012). Big data: the management revolution. Harvard business review, 90(10), 60-68.

2. Binette O, Steorts RC. (Almost) all of entity resolution. Science Advances. 2022;8(12):eabi8021.

3. Chen C, Frey CB, Presidente G. Privacy regulation and firm performance: Estimating the GDPR effect globally. The Oxford Martin Working Paper Series on Technological and Economic Change; 2022.
4. Jonas J. Senzing CEO Jeff Jonas on Entity Resolution in Data Analytics. https://www.youtube.com/watch?v=bNmdzEyb9ac; October 25, 2023.