Archives | Мегапьютер

Add support for stop list dictionaries when analyzing spelling errors

15.05.2018Yuri Slynko

In previous builds of PolyAnalyst the only way to stop a word from showing up in the spell checker was to add it to the morphology dictionary. This would sometimes result in having to add things to the morphology dictionary that might not really belong there, such as product codes (Model ACBXYZ) and the occasional single foreign word (e.g. yukata). In order to prevent this dictionary mismatch PolyAnalyst now includes stoplist functionality for spellchecking. In other words you can define a list of words for the spellchecker to ignore without assigning them as actual English words. Without a stop list: With a stop list: no more «yukata»!

Manual validation of entities in Entity Extraction

15.05.2018Yuri Slynko

Algorithmic extraction of entities from text is a powerful tool and a core feature of PolyAnalyst, but it can be difficult to get results with absolutely no false positives. Previously, the user would have to reword the algorithm little by little to remove each false positive, or find some other way to filter the results. Fortunately PolyAnalyst supports manual validation of entity extraction results. In effect this means that a user can mark each extraction as being invalid, valid or null, and only those that are not invalid will make their way into the actual dataset. This has a number of advantages beyond not needing to write additional Extended Pattern Definition Language (XPDL) code. For one, it makes it easier for multiple users to collaborate and manipulate the results. Additionally, other users can see what types of extractions are being marked as invalid, which may give them a deeper understanding…

Creating a pre-populated dictionary directly from the results of an analysis

22.08.2013Yuri Slynko

PolyAnalyst 6.5.1402 introduces a new, convenient feature for creating custom dictionaries. When analyzing text, you can use dictionaries to bias and tweak your analytic results, such as by excluding keywords through the use of a stop list, or grouping keywords as synonyms through the use of a thesaurus-like dictionary. PolyAnalyst ships with a variety of default dictionaries to kick start your analytical projects. Sometimes the default dictionaries are enough, but sometimes they are insufficient, in which case you are then faced with the task of creating a new dictionary. This often happens because of the domain of your analytical task. To really gain insight from the data, you need to teach PolyAnalyst about the jargon the speakers in the data happen to be using. Just like visiting a foreign country, you can get around town much more easily when you speak the local language. The traditional way of doing this…

PolyAnalyst incorporates WordNet 3.0

06.04.2009Yuri Slynko

For several years PolyAnalyst has successfully used WordNet 1.6 to provide customers with a basic starter English dictionary of terms and relations. WordNet is a popular online dictionary developed and maintained by Princeton University. PolyAnalyst incorporates a select number of word properties from WordNet into its various natural language processing tools. For example, PolyAnalyst uses WordNet to flesh out lists of hypernyms, hyponyms, merynyms, and antonyms. PolyAnalyst 6.0.950 introduces support for WordNet 3.0. After upgrading you will notice the new dictionary in the dictionaries list displayed by the Manage Dictionaries window. All text processing operations you perform after the upgrade will use the new dictionary version by default. An example of the new dictionaries window: For more information on WordNet, check out http://wordnet.princeton.edu/. To learn more about PolyAnalyst, visit https://www.megaputer.com/polyanalyst.

PolyAnalyst 6 adds a simple way of grouping several boolean columns into a few categorical columns

06.04.2009Yuri Slynko

Starting in PolyAnalyst 6.0.950, you now have access to a new node named Categorize Binaries. The node provides a method for deriving a compact representation of hundreds of multiple choice variables. The node operates by converting a large set of binary variables into a small number of categorical variables. For example, suppose you have a dataset of sales transactions involving hundreds of products. Suppose the dataset is designed so that each separate product is represented in a separate column, where a product column is true if the product was present in a sales transaction. Let us take a popular product line like names of beer (Pilzner, Carlsberg, Guinness, etc). Customer Id Bought Pilzner? Bought Carlsberg? Bought Guinness? 1 Yes No No 2 No No Yes 3 No Yes No For most of the transactions only one brand is selected. In other words, most of the binary product dataset attributes are…

Megaputer ships PolyAnalyst 4.5

09.04.2002Sergei Ananyan

The world's first system for joint analysis of structured data and text Bloomington, IN - April 9, 2002. -- Megaputer Intelligence Inc. announced the release of PolyAnalyst 4.5, the world's first Data Mining application capable of efficiently analyzing both structured data and text within a single powerful system. Based on years of delivering analytical data and text mining solutions, Megaputer continues to prove its leadership in the analytical software market by merging its technologies into a single platform to serve a new growing demand in the marketplace. The implementation of text mining capabilities empowers users of PolyAnalyst 4.5 to address new classes of tasks such as the analysis of customer correspondence, insurance claims, and call center notes. PolyAnalyst provides tools for semantic analysis, both supervised and unsupervised, of unstructured natural language text held in relational databases. "Over 80% of an organization's knowledge is scattered through unstructured and semi-structured data. In...

Megaputer Ships PolyAnalyst 4.4

03.04.2001Sergei Ananyan

Applying models to external data through a standard SQL-based protocol Bloomington, IN - April 3, 2001. -- Megaputer Intelligence Inc. announced the release of PolyAnalyst 4.4, a high-end Data Mining tool offering new features that dramatically simplify the process of integrating the results obtained through PolyAnalyst data analysis in any external decision support applications and help processing very large databases in their entirety. The new PolyAnalyst Model Application Wizard allows simple scoring of data in any external source through a standard SQL-based protocol, OLE DB. PolyAnalyst can export created models in XML/PMML format or store these models in a new OLE DB for Data Mining format suitable for direct data scoring. Additionally, PolyAnalyst 4.4 provides In-Place Data Mining capability for working with very large data in dynamic SQL mode through an intuitive interface. "Megaputer is proud to ship the industry's first data mining product capable of training on very large...