Megaputer Blog | Мегапьютер

Fuzzy Matching — Comparing Records With String Distance Measures

24/03/2020Chris Farris, Data Mining

In previous posts we have loosely discussed the act of comparing strings. Until now this has been a nebulous term relying primarily on intuition to understand but if we want automated systems such as PolyAnalyst to perform what we call comparison it needs to be well defined and based on calculation. This concept is typically referred to as a string metric and is a method of putting into numbers how similar two strings are. Despite the name, many of these measurements are not in fact metrics by definition because they do not all obey the triangle inequality; however, they remain very useful in the measurement of strings. There are many string metrics of varying degree of complexity and usefulness. A traditional exemplar of string metrics is the Levenshtein distance. The Levenshtein distance can be informally defined as the number of edits it takes to turn one string into the other…

Fuzzy Matching

18/02/2020Chris Farris, Data Mining, Featured

In our daily work we often need to make comparisons of data between and within datasets. Such operations including looking for multiple copies of records for elimination or some other analysis task. Another type of operation is a “join” which is a method of combining two tables into one. For example, we may have one table that contains products and their prices while another table contains products and the current number of orders for them. To predict anticipated revenue we really need one table with all of that information at once and a join operation on the tables can produce such a table. These types of procedures may seem rudimentary and common and for the most part this is true. If our datasets contain unique keys for each record then these tasks are simple and can be conducted without much thought. However, there are many scenarios in which we do…

Challenges in Human Name Detection

22/01/2020Margaret Glide, Text Analytics

When working with big data, it is often possible to automate the slow, mind-numbing information extraction tasks, for precisely the reason that they require little creative energy. One such automatable process is detecting, extracting, and resolving human names from text data. For example, an insurance company may want to automatically extract from call center notes the names of all parties involved in an accident. However, they process hundreds of thousands of records each week. While a person may look at a single text and easily recognize names, it can be time-consuming or nearly impossible to examine each record individually and compile them into a suitable output. Using algorithms that people apply unthinkingly to detect names, we can teach software to do this time-consuming work for us. Some Approaches and Challenges Building Name Dictionaries. The simplest approach to detecting names in big data is to build dictionaries, or lists, of common…