Text Analytics Archives | Мегапьютер

Модель прогноза распространения коронавируса размещена на карте технологий ICT.Moscow

30/05/2020Дмитрий Гольцов

Модель прогноза распространения коронавируса отмечена на карте технологий ICT.Moscow 30 мая 2020 Разработанная специалистами Мегапьютер прогнозная модель развития коронавирусной инфекции помещена на Карту COVID-Tech практик в России, подготовленную порталом #ICT.Moscow. Сейчас на карте (версия 1.1) отмечено 80 российских практик в области COVID-Tech, они разделены на шесть категорий: диагностика COVID-19 и скрининги (13 кейсов); поиск препаратов (2); платформы данных, аналитика, базы знаний (16); контроль распространения и социального дистанцирования (27 + 3 в версии 1.1); информационная поддержка (14); умное оборудование и роботы (5). Модель представлена в разделе Платформы данных / Аналитика / Базы знаний. Карта российских практик, ориентированных на борьбу с коронавирусом. ICT.Moscow Также ICT.Moscow опубликовал материал о достижениях российских компаний в борьбе с пандемией COVID-19. В материале приведены мнения представителей отечественной IT-отрасли о вкладе российских технологичных решений в борьбу с коронавирусной инфекцией и как этот вклад оценивается в мире. Компания Мегапьютер представила свое видение обсуждаемой ситуации: «По нашему мнению, отечественные разработчики продемонстрировали внушительную продуктивность…

Специалисты Мегапьютер Интеллидженс разработали модель прогноза пиков заболеваемости COVID-19 в регионах России

20/05/2020Дмитрий Гольцов

Специалисты Мегапьютер Интеллидженс разработали модель прогноза пиков заболеваемости COVID-19 в регионах России 20 мая 2020 РАБОЧАЯ МОДЕЛЬ ПРОЕКТА ВЫЛОЖЕНА В ОТКРЫТЫЙ ДОСТУП Ознакомиться ЦЕЛЬ ПРОЕКТА В условиях пандемии вирусной инфекции COVID-19 ключевыми проблемами на уровне государства являются опасность критической перегрузки национальной системы здравоохранения и риск глубокого экономического спада, вызванного массовым закрытием предприятий малого и среднего бизнеса. Основной задачей модели является повышение информированности управленческих решений. Надежный прогноз развития картины заболеваемости дает возможность органам государственной власти, медицинским структурам и коммерческим организациям оперативно определять масштаб потенциальных угроз. Имея данные о сроках активного роста заболеваемости, становится возможным заблаговременно предпринять меры по снижению негативного воздействия и оперативно перераспределить ресурсы. ОПИСАНИЕ МОДЕЛИ Для каждого региона России имеется собственный ежедневный прогноз количества подтвержденных и активных случаев заболевания, числа выздоровлений и летальных исходов. В результате анализа данных о пандемии алгоритмами машинного обучения была получена статистическая картина течения и распространения заболевания. По каждому показателю алгоритм формирует два варианта развития событий:…

Normalization in Preparation for Fuzzy Matching

02/04/2020Chris Farris

When we need to compare entities stored as string data we often need to consider many different pieces of the string. For example, suppose we examine the following names in our data: Steven M Johnson Steve Michael Johnson When we, as humans, compare these entities, we don’t directly compare each string as a whole. Instead we compare substrings. “Steven” is compared to “Steve,” “M” to “Michael,” and “Johnson” to “Johnson.” We do this because we recognize that names aren’t one complete thing but comprised of smaller attributes – in this case the first, middle, and last names. It logically makes more sense to compare these substrings together rather than the entire string. “Steven M Johnson” as a whole may not be very similar (by whichever way we define that) to “Steve Michael Johnson” but the individual components are not that far off. This process of deconstructing entities to more atomic…

Challenges in Human Name Detection

22/01/2020Margaret Glide

When working with big data, it is often possible to automate the slow, mind-numbing information extraction tasks, for precisely the reason that they require little creative energy. One such automatable process is detecting, extracting, and resolving human names from text data. For example, an insurance company may want to automatically extract from call center notes the names of all parties involved in an accident. However, they process hundreds of thousands of records each week. While a person may look at a single text and easily recognize names, it can be time-consuming or nearly impossible to examine each record individually and compile them into a suitable output. Using algorithms that people apply unthinkingly to detect names, we can teach software to do this time-consuming work for us. Some Approaches and Challenges Building Name Dictionaries. The simplest approach to detecting names in big data is to build dictionaries, or lists, of common…

What is Sentiment Analysis?

18/11/2019Elli Bourlai

Sentiment Analysis is another buzzword that we hear more and more often, but may not entirely understand what it does or how to use it. Sentiment Analysis application in business offers benefits to both companies and customers, such as improving products and services, identifying strengths and weaknesses of the competition, as well as targeted advertising. Sentiment Analysis, also known as Sentiment Detection or Opinion Mining, is the classification of the sentiment polarity in a given text that may express people’s opinions, appraisals, attitudes, and emotions toward entities, individuals, issues, events, topics and their attributes[1]. Put simply, it answers the question “how does the writer feel about this topic?” with the answer positive or negative, but may also include neutral. In other words, Sentiment Analysis identifies words or patterns in language that carry positive, negative, or neutral sentiment. As humans, we interpret sentiment based on not only our knowledge of the…

Add support for stop list dictionaries when analyzing spelling errors

15/05/2018Jeff Palan

In previous builds of PolyAnalyst the only way to stop a word from showing up in the spell checker was to add it to the morphology dictionary. This would sometimes result in having to add things to the morphology dictionary that might not really belong there, such as product codes (Model ACBXYZ) and the occasional single foreign word (e.g. yukata). In order to prevent this dictionary mismatch PolyAnalyst now includes stoplist functionality for spellchecking. In other words you can define a list of words for the spellchecker to ignore without assigning them as actual English words. Without a stop list: With a stop list: no more «yukata»!

Manual validation of entities in Entity Extraction

15/05/2018Jeff Palan

Algorithmic extraction of entities from text is a powerful tool and a core feature of PolyAnalyst, but it can be difficult to get results with absolutely no false positives. Previously, the user would have to reword the algorithm little by little to remove each false positive, or find some other way to filter the results. Fortunately PolyAnalyst supports manual validation of entity extraction results. In effect this means that a user can mark each extraction as being invalid, valid or null, and only those that are not invalid will make their way into the actual dataset. This has a number of advantages beyond not needing to write additional Extended Pattern Definition Language (XPDL) code. For one, it makes it easier for multiple users to collaborate and manipulate the results. Additionally, other users can see what types of extractions are being marked as invalid, which may give them a deeper understanding…

Why We Use Dependency Parsing

15/05/2018Rebecca Hale

For beginners, text analysis often begins with a simple keyword search. Analysts can start by searching for “great service”. They may then begin adding other searches like “service was great” or “service is so great”. This, of course, is a very rudimentary approach that will miss a lot of relevant comments, so users who want to get more results quickly will advance to proximity searches. For example, they may search for “great” within three words of “service”. While some may use word lists (“awesome”, “good”, or “excellent” service) or parts of speech (adjectives near “service”) to improve results, most text analysis systems stop right there. For advanced text searching, however, you need to have information about word relationships. For example, “the insured driver hit a bus” and “a bus hit an insured driver” are two very different events that might help a text analysis system determine who was at fault…

Prepping the XPDL Seminar Series

09/05/2018Rebecca Hale

Registration for the XPDL webinar series opens June 1, 2018. Following numerous requests from our customers for more XPDL training material, I’ve been developing an XPDL seminar series. We receive a lot of questions about how best to use XPDL to extract information from text, because it is such an important task for text analysis. Data analysts, survey conductors, customer experience managers… anyone working with text data needs a way to extract the information they need. After testing these presentations on some of our analysts, we’ll begin preparing brand-new training modules for our customers. This week, we had a lot of great questions from our analysts, making sure they understand the more complex rule types. Practicing the series at our weekly meetings has given us a lot of great tips and improvements— for example, we learned that users want lots of practical text examples, showing exactly the effect of the…

Entity Resolution in Text Analysis

02/04/2018Chris Farris

In our analysis of textual data there are many different types of entities we need to identify and capture such as companies, organizations, places, currencies, stocks, and people. This task alone can be challenging, but in many scenarios it is further complicated by the fact that entities can be highly contextual and can be referenced in a myriad of forms. Referencing people easily demonstrates this phenomenon. For example, the individual whose name is Steve Michael Johnson may be referred to as Steve, Steve Johnson, Mr. Johnson, and in other ways. However, this is only the beginning of the complexity. All of these references are simply versions of the full name and so they are not too difficult to link together. However, depending on the context, an individual may be referenced in a manner using no fragment of their human name. Consider auto accident reports. In this data an individual may…