Text Analytics Archives - Мегапьютер Интеллидженс

Entity Resolution in Text Analysis

26/05/2018Chris Farris

In our analysis of textual data there are many different types of entities we need to identify and capture such as companies, organizations, places, currencies, stocks, and people. This task alone can be challenging, but in many scenarios it is further complicated by the fact that entities can be highly contextual and can be referenced in a myriad of forms. Referencing people easily demonstrates this phenomenon. For example, the individual whose name is Steve Michael Johnson may be referred to as Steve, Steve Johnson, Mr. Johnson, and in other ways. However, this is only the beginning of the complexity. All of these references are simply versions of the full name and so they are not too difficult to link together. However, depending on the context, an individual may be referenced in a manner using no fragment of their human name. Consider auto accident reports. In this data an individual may…

Normalization in Preparation for Fuzzy Matching

24/05/2018Chris Farris

When we need to compare entities stored as string data we often need to consider many different pieces of the string. For example, suppose we examine the following names in our data: Steven M Johnson Steve Michael Johnson When we, as humans, compare these entities, we don’t directly compare each string as a whole. Instead we compare substrings. “Steven” is compared to “Steve,” “M” to “Michael,” and “Johnson” to “Johnson.” We do this because we recognize that names aren’t one complete thing but comprised of smaller attributes – in this case the first, middle, and last names. It logically makes more sense to compare these substrings together rather than the entire string. “Steven M Johnson” as a whole may not be very similar (by whichever way we define that) to “Steve Michael Johnson” but the individual components are not that far off. This process of deconstructing entities to more atomic…

Challenges in Human Name Detection

22/05/2018Margaret Glide

When working with big data, it is often possible to automate the slow, mind-numbing information extraction tasks, for precisely the reason that they require little creative energy. One such automatable process is detecting, extracting, and resolving human names from text data. For example, an insurance company may want to automatically extract from call center notes the names of all parties involved in an accident. However, they process hundreds of thousands of records each week. While a person may look at a single text and easily recognize names, it can be time-consuming or nearly impossible to examine each record individually and compile them into a suitable output. Using algorithms that people apply unthinkingly to detect names, we can teach software to do this time-consuming work for us. Some Approaches and Challenges Building Name Dictionaries. The simplest approach to detecting names in big data is to build dictionaries, or lists, of common…

What is Sentiment Analysis?

18/05/2018Elli Bourlai

Sentiment Analysis is another buzzword that we hear more and more often, but may not entirely understand what it does or how to use it. Sentiment Analysis application in business offers benefits to both companies and customers, such as improving products and services, identifying strengths and weaknesses of the competition, as well as targeted advertising. Sentiment Analysis, also known as Sentiment Detection or Opinion Mining, is the classification of the sentiment polarity in a given text that may express people’s opinions, appraisals, attitudes, and emotions toward entities, individuals, issues, events, topics and their attributes[1]. Put simply, it answers the question “how does the writer feel about this topic?” with the answer positive or negative, but may also include neutral. In other words, Sentiment Analysis identifies words or patterns in language that carry positive, negative, or neutral sentiment. As humans, we interpret sentiment based on not only our knowledge of the…

Add support for stop list dictionaries when analyzing spelling errors

15/05/2018Jeff Palan

In previous builds of PolyAnalyst the only way to stop a word from showing up in the spell checker was to add it to the morphology dictionary. This would sometimes result in having to add things to the morphology dictionary that might not really belong there, such as product codes (Model ACBXYZ) and the occasional single foreign word (e.g. yukata). In order to prevent this dictionary mismatch PolyAnalyst now includes stoplist functionality for spellchecking. In other words you can define a list of words for the spellchecker to ignore without assigning them as actual English words. Without a stop list: With a stop list: no more “yukata”!

Manual validation of entities in Entity Extraction

15/05/2018Jeff Palan

Algorithmic extraction of entities from text is a powerful tool and a core feature of PolyAnalyst, but it can be difficult to get results with absolutely no false positives. Previously, the user would have to reword the algorithm little by little to remove each false positive, or find some other way to filter the results. Fortunately PolyAnalyst supports manual validation of entity extraction results. In effect this means that a user can mark each extraction as being invalid, valid or null, and only those that are not invalid will make their way into the actual dataset. This has a number of advantages beyond not needing to write additional Extended Pattern Definition Language (XPDL) code. For one, it makes it easier for multiple users to collaborate and manipulate the results. Additionally, other users can see what types of extractions are being marked as invalid, which may give them a deeper understanding…

Why We Use Dependency Parsing

15/05/2018Rebecca Hale

For beginners, text analysis often begins with a simple keyword search. Analysts can start by searching for “great service”. They may then begin adding other searches like “service was great” or “service is so great”. This, of course, is a very rudimentary approach that will miss a lot of relevant comments, so users who want to get more results quickly will advance to proximity searches. For example, they may search for “great” within three words of “service”. While some may use word lists (“awesome”, “good”, or “excellent” service) or parts of speech (adjectives near “service”) to improve results, most text analysis systems stop right there. For advanced text searching, however, you need to have information about word relationships. For example, “the insured driver hit a bus” and “a bus hit an insured driver” are two very different events that might help a text analysis system determine who was at fault…

Prepping the XPDL Seminar Series

09/05/2018Rebecca Hale

Registration for the XPDL webinar series opens June 1, 2018. Following numerous requests from our customers for more XPDL training material, I’ve been developing an XPDL seminar series. We receive a lot of questions about how best to use XPDL to extract information from text, because it is such an important task for text analysis. Data analysts, survey conductors, customer experience managers… anyone working with text data needs a way to extract the information they need. After testing these presentations on some of our analysts, we’ll begin preparing brand-new training modules for our customers. This week, we had a lot of great questions from our analysts, making sure they understand the more complex rule types. Practicing the series at our weekly meetings has given us a lot of great tips and improvements— for example, we learned that users want lots of practical text examples, showing exactly the effect of the…

PolyAnalyst Improves Patient Safety at the U.S. Dept. of Veterans Affairs

12/02/2008Sergei Ananyan

February 12, 2008. The National Center for Patient Safety (NCPS) of the Department of Veterans Affairs (VA) implements text mining technologies to enhance the analysis of patient safety reports. The VA NCPS analysts are utilizing the PolyAnalyst™ data and text mining system from Megaputer Intelligence to detect common patterns, identify emerging trends, and determine root causes of issues in patient safety reports received from the 153 hospitals operated by the VA. The goal of the VA NCPS is the reduction and prevention of inadvertent harm to patients as a result of care. To accomplish this goal, the VA analysts strive to learn from safety reports representing close calls, also known as “near misses,” which occur at a much higher frequency than actual adverse events. This approach focuses efforts on continually identifying potential problems and proactively fixing them to improve patient safety and quality of care. The most informative part of…

Discovering Hidden Threats Through Data and Text Mining of Historical Safety Reports

01/02/2004Sergei Ananyan

Sergei Ananyan speaks about "Discovering Hidden Threats Through Data and Text Mining of Historical Safety Reports" When: February 25-26, 2004 Where: Walt Disney World, Florida View the Homeland Security Conference Website Modern analytical technologies of data and text mining have opened new opportunities for better safety management through automated intelligent analysis of available safety reports. Advanced analysis of historical safety reports can reveal important hidden information about existing vulnerabilities and threats. This information arms safety officer with key knowledge necessary for easier prevention of future security breakdowns. This presentation will review the methodology, analytical tools and results of recent flight safety data analysis projects carried out for Federal Aviation Administration, Southwest Airlines and International Air Traffic Association.