Megaputer Blog - Мегапьютер Интеллидженс

Entity Resolution in Text Analysis

26/05/2018Chris Farris, Data Mining, Text Analytics

In our analysis of textual data there are many different types of entities we need to identify and capture such as companies, organizations, places, currencies, stocks, and people. This task alone can be challenging, but in many scenarios it is further complicated by the fact that entities can be highly contextual and can be referenced in a myriad of forms. Referencing people easily demonstrates this phenomenon. For example, the individual whose name is Steve Michael Johnson may be referred to as Steve, Steve Johnson, Mr. Johnson, and in other ways. However, this is only the beginning of the complexity. All of these references are simply versions of the full name and so they are not too difficult to link together. However, depending on the context, an individual may be referenced in a manner using no fragment of their human name. Consider auto accident reports. In this data an individual may…

Normalization in Preparation for Fuzzy Matching

24/05/2018Chris Farris, Text Analytics

When we need to compare entities stored as string data we often need to consider many different pieces of the string. For example, suppose we examine the following names in our data: Steven M Johnson Steve Michael Johnson When we, as humans, compare these entities, we don’t directly compare each string as a whole. Instead we compare substrings. “Steven” is compared to “Steve,” “M” to “Michael,” and “Johnson” to “Johnson.” We do this because we recognize that names aren’t one complete thing but comprised of smaller attributes – in this case the first, middle, and last names. It logically makes more sense to compare these substrings together rather than the entire string. “Steven M Johnson” as a whole may not be very similar (by whichever way we define that) to “Steve Michael Johnson” but the individual components are not that far off. This process of deconstructing entities to more atomic…

Fuzzy Matching — Comparing Records With String Distance Measures

24/05/2018Chris Farris, Data Mining

In previous posts we have loosely discussed the act of comparing strings. Until now this has been a nebulous term relying primarily on intuition to understand but if we want automated systems such as PolyAnalyst to perform what we call comparison it needs to be well defined and based on calculation. This concept is typically referred to as a string metric and is a method of putting into numbers how similar two strings are. Despite the name, many of these measurements are not in fact metrics by definition because they do not all obey the triangle inequality; however, they remain very useful in the measurement of strings. There are many string metrics of varying degree of complexity and usefulness. A traditional exemplar of string metrics is the Levenshtein distance. The Levenshtein distance can be informally defined as the number of edits it takes to turn one string into the other…