Data Mining Archives - Мегапьютер Интеллидженс

Entity Resolution in Text Analysis

26/05/2018Chris Farris

In our analysis of textual data there are many different types of entities we need to identify and capture such as companies, organizations, places, currencies, stocks, and people. This task alone can be challenging, but in many scenarios it is further complicated by the fact that entities can be highly contextual and can be referenced in a myriad of forms. Referencing people easily demonstrates this phenomenon. For example, the individual whose name is Steve Michael Johnson may be referred to as Steve, Steve Johnson, Mr. Johnson, and in other ways. However, this is only the beginning of the complexity. All of these references are simply versions of the full name and so they are not too difficult to link together. However, depending on the context, an individual may be referenced in a manner using no fragment of their human name. Consider auto accident reports. In this data an individual may…

Fuzzy Matching – Comparing Records With String Distance Measures

24/05/2018Chris Farris

In previous posts we have loosely discussed the act of comparing strings. Until now this has been a nebulous term relying primarily on intuition to understand but if we want automated systems such as PolyAnalyst to perform what we call comparison it needs to be well defined and based on calculation. This concept is typically referred to as a string metric and is a method of putting into numbers how similar two strings are. Despite the name, many of these measurements are not in fact metrics by definition because they do not all obey the triangle inequality; however, they remain very useful in the measurement of strings. There are many string metrics of varying degree of complexity and usefulness. A traditional exemplar of string metrics is the Levenshtein distance. The Levenshtein distance can be informally defined as the number of edits it takes to turn one string into the other…

Fuzzy Matching

24/05/2018Chris Farris

In our daily work we often need to make comparisons of data between and within datasets. Such operations including looking for multiple copies of records for elimination or some other analysis task. Another type of operation is a “join” which is a method of combining two tables into one. For example, we may have one table that contains products and their prices while another table contains products and the current number of orders for them. To predict anticipated revenue we really need one table with all of that information at once and a join operation on the tables can produce such a table. These types of procedures may seem rudimentary and common and for the most part this is true. If our datasets contain unique keys for each record then these tasks are simple and can be conducted without much thought. However, there are many scenarios in which we do…

Improving how mixed sources of data are accurately merged together through the use of fuzzy joins

21/05/2018Chris Farris

In our daily work we often need to combine two or more datasets together into one. This type of operation, known as a join, is rather simple when each record contains a unique ID present in both datasets. However, there are many scenarios where datasets use different methods of creating unique keys and thus do not match or do not have unique keys at all. In these situations the traditional join operation does not suffice. For example, we have many projects involving the analysis of individual people. One dataset may be from one source such as a hospital which will contain medical data for that person while another dataset may be from another source such as an insurance company which will contain policy information. It is unlikely that these two institutions share the same record keeping system in which real world individuals are given the same unique key in both…

Discovering Hidden Threats Through Data and Text Mining of Historical Safety Reports

01/02/2004Sergei Ananyan

Sergei Ananyan speaks about "Discovering Hidden Threats Through Data and Text Mining of Historical Safety Reports" When: February 25-26, 2004 Where: Walt Disney World, Florida View the Homeland Security Conference Website Modern analytical technologies of data and text mining have opened new opportunities for better safety management through automated intelligent analysis of available safety reports. Advanced analysis of historical safety reports can reveal important hidden information about existing vulnerabilities and threats. This information arms safety officer with key knowledge necessary for easier prevention of future security breakdowns. This presentation will review the methodology, analytical tools and results of recent flight safety data analysis projects carried out for Federal Aviation Administration, Southwest Airlines and International Air Traffic Association.