The present invention presents embodiments of methods, systems, and computer-readable media for the retrieval, mining, filtering and visualization of information stored on a plural of computers connected to the Internet and on a local computer. Embodiments of this invention generate a conceptual search query using a description provided by a user, perform user selectable conceptual filtering of search results, concept following and link following to expand search results, search for files that may or may not contain certain information, rank concepts contained in search results or one or more files, compute relevancy rank of a file in search results, use conceptual path maps to display logic or statistical relationships among search results, monitor changes in information in a search or a file, and protect files or searches based on information contents.

Internet and computer information retrieval and mining with intelligent conceptual filtering, visualization and automation

A data processing system includes one or more nodes, each node including a memory sub-system. The sub-system includes a fine-grained, memory, and a less-fine-grained (e.g., page-based) memory. The fine-grained memory optionally serves as a cache and/or as a write buffer for the page-based memory. Software executing on the system uses a node address space which enables access to the page-based memories of all nodes. Each node optionally provides ACID memory properties for at least a portion of the space. In at least a portion of the space, memory elements are mapped to locations in the page-based memory. In various embodiments, some of the elements are compressed, the compressed elements are packed into pages, the pages are written into available locations in the page-based memory, and a map maintains an association between the some of the elements and the locations.

System including a fine-grained memory and a less-fine-grained memory

Disclosed is an improved approach to implement I/O and storage device management in a virtualization environment. According to some approaches, a Service VM is employed to control and manage any type of storage device, including directly attached storage in addition to networked and cloud storage. The Service VM implements the Storage Controller logic in the user space, and can be migrated as needed from one node to another. IP-based requests are used to send I/O request to the Service VMs. The Service VM can directly implement storage and I/O optimizations within the direct data access path, without the need for add-on products.

Architecture for managing I/O and storage for a virtualization environment

Disclosed is an improved approach for using advanced metadata to implement an architecture for managing I/O operations and storage devices for a virtualization environment. According to some embodiments, a Service VM is employed to control and manage any type of storage device, including directly attached storage in addition to networked and cloud storage. The advanced metadata is used to track data within the storage devices. A lock-free approach is implemented in some embodiments to access and modify the metadata.

Metadata for managing I/O and storage for a virtualization environment

Disclosed is an approach for overseeing the overall state of a virtual storage system, and for taking actions as necessary to manage the health and efficient performance of that system. A curator entity is provided on a distributed basis to manage and perform these functions. A master curator manages this workload that is performed by multiple slave curators. MapReduce operations are performed to implement the curator workload. Periodically, the master curator coordinates a full scan of all the metadata in the system using three MapReduce operations. Partial scans may also be run using a single MapReduce operation.

Method and system for implementing a maintenance service for managing I/O and storage for a virtualization environment

A method and system for backing up primary nodes onto backup nodes where the primary nodes can each originate a distributed transaction and can participate in a distributed transaction. The backup nodes, after a primary node failure, undo all transactions whose state is unknown on the backup node and all committed transactions which, if kept, would lead to an inconsistent set of backup nodes. First, committed transactions are undone if commits for them were not received on all back up nodes to the primary nodes that participated in the transaction. Second, all committed transactions that potentially depend on the committed transactions that were undone are also undone. Only those transactions that are provably independent of the committed transactions that were undone are kept. The result is a set of back up nodes which can support operations until the failing primary node is restored.

System and method for replication of distributed databases that span multiple primary nodes

A method of processing a transaction includes processing a transaction workload in a primary process pair on a first node in a cluster of nodes, the processing using at least one stable storage volume for storing a database and another stable storage volume for storing a log, the at least one stable storage volume and the log storage volume forming a log storage group. The method further includes performing checkpointing operations via the network from the primary process pair to a backup process pair while processing the transaction workload, the backup process pair operating on a second node in the cluster of nodes. The method further includes detecting a failure making the first node inoperable or inaccessible, and after detecting the failure, engaging the backup process pair to take over the transaction processing workload of the primary process pair, the backup process pair being configured to operate with the log storage group used by the primary process pair on the failed node.

Transaction processing apparatus and method

A method of transferring a transaction workload of a primary process pair. The primary pair has access to a stable storage volume for storing data items and a log for storing transaction processing updates to data items. A backup process pair is established for the primary pair and information is transferred thereto so that it can take over for the primary pair should that pair fail. Additionally, a lock table is maintained and update records are written to the log by the primary pair. The update records include any locks from the lock table that related to the update records. When a failure of the primary pair occurs, the storage volume is locked, and the backup pair reads the log to reconstruct a lock table. Once the locks have been reconstructed, the lock on the storage volume is released and the backup pair operates to process the workload.

Minimum latency reinstatement of database transaction locks

A remote data facility (RDF) capable of performing a lockstep data replication procedure (“LockStep Procedure”). When the LockStep Procedure is invoked, and when an application program has committed a transaction, the application program is prevented from executing other procedures until the application is notified that audit records associated with that transaction have been safely stored to the backup system. Since the application program is prevented from executing other procedures, no decision based on the commit will be made until after the application is notified that all the audit records associated with the transaction are safely stored in the backup system.

Method and apparatus for lockstep data replication

HP's Store All with Express Query is a scalable commercial file archiving product that offers sophisticated file metadata management and search capabilities [3]. A new REST API enables fast, efficient searching to find all files that meet a given set of metadata criteria and the ability to tag files with custom metadata fields. The product brings together two significant systems: a scale out file system and a metadata database based on LazyBase [10]. In designing and building the combined product, we identified several real-world issues in using a pipelined database system in a distributed environment, and overcame several interesting design challenges that were not contemplated by the original research prototype. This paper highlights our experiences.

/pdf/from-research-to-practice-experiences-engineering-a-mh8xyi3y26.pdf

Charles Stuart Johnson

Papers

System and method for replication of distributed databases that span multiple primary nodes

Transaction processing apparatus and method

Minimum latency reinstatement of database transaction locks

Method and apparatus for lockstep data replication

From research to practice: experiences engineering a production metadata database for a scale out file system