Megaputer Blog - Мегапьютер Интеллидженс

Fuzzy Matching

24/05/2018Chris Farris, Data Mining, Featured

In our daily work we often need to make comparisons of data between and within datasets. Such operations including looking for multiple copies of records for elimination or some other analysis task. Another type of operation is a “join” which is a method of combining two tables into one. For example, we may have one table that contains products and their prices while another table contains products and the current number of orders for them. To predict anticipated revenue we really need one table with all of that information at once and a join operation on the tables can produce such a table. These types of procedures may seem rudimentary and common and for the most part this is true. If our datasets contain unique keys for each record then these tasks are simple and can be conducted without much thought. However, there are many scenarios in which we do…

Challenges in Human Name Detection

22/05/2018Margaret Glide, Text Analytics

When working with big data, it is often possible to automate the slow, mind-numbing information extraction tasks, for precisely the reason that they require little creative energy. One such automatable process is detecting, extracting, and resolving human names from text data. For example, an insurance company may want to automatically extract from call center notes the names of all parties involved in an accident. However, they process hundreds of thousands of records each week. While a person may look at a single text and easily recognize names, it can be time-consuming or nearly impossible to examine each record individually and compile them into a suitable output. Using algorithms that people apply unthinkingly to detect names, we can teach software to do this time-consuming work for us. Some Approaches and Challenges Building Name Dictionaries. The simplest approach to detecting names in big data is to build dictionaries, or lists, of common…

Improving how mixed sources of data are accurately merged together through the use of fuzzy joins

21/05/2018Chris Farris, Data Mining

In our daily work we often need to combine two or more datasets together into one. This type of operation, known as a join, is rather simple when each record contains a unique ID present in both datasets. However, there are many scenarios where datasets use different methods of creating unique keys and thus do not match or do not have unique keys at all. In these situations the traditional join operation does not suffice. For example, we have many projects involving the analysis of individual people. One dataset may be from one source such as a hospital which will contain medical data for that person while another dataset may be from another source such as an insurance company which will contain policy information. It is unlikely that these two institutions share the same record keeping system in which real world individuals are given the same unique key in both…