A phrase discovery is a method of identifying sequences of terms in a database. First, a selection of one or more relevant sequences of terms, such as relevant text, is provided. Next, several shorter sequences of terms, such as phrases, are extracted from the provided relevant sequences of terms. The extracted sequences of terms are then reduced through a culling process. A gathering process then emphasizes the more relevant of the extracted and culled sequences of terms and de-emphasizes the more generic of the extracted and culled sequences of terms. The gathering process can also include iteratively retrieving additional selections of relevant sequences (e.g., text), extracting and culling additional sequences of terms (e.g., phrases), emphasizing and de-emphasizing extracted and culled sequences of terms and accumulating all gathered sequences of terms. The resulting gathered sequences of terms are then output.

System, method and apparatus for discovering phrases in a database

The invention is a method, system and computer program for automatically discovering concepts from a corpus of documents and automatically generating a labeled concept hierarchy. The method involves extraction of signatures from the corpus of documents. The similarity between signatures is computed using a statistical measure. The frequency distribution of signatures is refined to alleviate any inaccuracy in the similarity measure. The signatures are also disambiguated to address the polysemy problem. The similarity measure is recomputed based on the refined frequency distribution and disambiguated signatures. The recomputed similarity measure reflects actual similarity between signatures. The recomputed similarity measure is then used for clustering related signatures. The signatures are clustered to generate concepts and concepts are arranged in a concept hierarchy. The concept hierarchy automatically generates query for a particular concept and retrieves relevant documents associated with the concept.

System and method for automatically discovering a hierarchy of concepts from a corpus of documents

A document categorisation system, including a clusterer for generating clusters of related electronic documents based on features extracted from said documents, and a filter module for generating a filter on the basis of said clusters to categorise further documents received by said system. The system may include an editor for manually browsing and modifying the clusters. The categorisation of the documents is based on n-grams, which are used to determine significant features of the documents. The system includes a trend analyzer for determining trends of changing document categories over time, and for identifying novel clusters. The system may be implemented as a plug-in module for a spreadsheet application, providing a convenient means for one-off or ongoing analysis of text entries in a worksheet.

A document categorisation system

A system and method for efficiently generating cluster groupings in a multi-dimensional concept space is described. A plurality of terms are extracted from each document in a collection of stored unstructured documents. A concept space is built over the document collection. Terms substantially correlated between a plurality of documents within the document collection are identified. Each correlated term is expressed as a vector mapped along an angle θ originating from a common axis in the concept space. A difference between the angle θ for each document and an angle σ for each cluster within the concept space is determined. Each such cluster is populated with those documents having such difference between the angle θ for each such document and the angle σ for each such cluster falling within a predetermined variance. A new cluster is created within the concept space those documents having such difference between the angle θ for each such document and the angle σ for each such cluster falling outside the predetermined variance.

System and method for efficiently generating cluster groupings in a multi-dimensional concept space

A method of viewing information associated with data in a spreadsheet, includes providing a document including data and information associated with the data, parsing the document to retrieve the associated information, processing the associated information to break the associated information down into at least one sentence, categorizing the at least one sentence to determine whether the at least one sentence corresponds to at least one category in a taxonomy corresponding to the data, assigning an association strength to the categorized at least one sentence, the association strength indicating a likelihood that the categorized at least one sentence actually corresponds to the at least one category in the taxonomy, filtering the at least one categorized sentence based on the association strength to determine whether to match the categorized at least one sentence with the at least one category in the taxonomy and outputting only the categorized at least one sentence matched with the at least one category in the taxonomy.

System and method for rendering data

An economic, scalable machine learning system and process perform document (concept) classification (210) with high accuracy using large topic schemes, including large hierarchical topic schemes. One or more highly relevant classification topics is suggested for a given document (concept) to be classified (210). The invention includes training (200) and concept classification (210) processes. The invention also provides methods that may be used as part of the training and/or concept classification processes, including: a method of scoring (303) the relevance of features in training concepts, a method of ranking concepts based on relevance score, and a method of voting on topics associated with an input concept. In a preferred embodiment, the invention is applied to the legal (case law) domain, classifying legal concepts (rules of law) according to a proprietary legal topic classification scheme (a hierarchical scheme of areas of law).

X. Allen Lu

Papers

System and method for classifying legal concepts using legal topic scheme

System und verfahren zur klassierung von juristischen konzepten unter verwendung eines juristischen themenschemas