In this paper, we present Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext. Google is designed to crawl and index the Web efficiently and produce much more satisfying search results than existing systems. The prototype with a full text and hyperlink database of at least 24 million pages is available at http://google.stanford.edu/. To engineer a search engine is a challenging task. Search engines index tens to hundreds of millions of web pages involving a comparable number of distinct terms. They answer tens of millions of queries every day. Despite the importance of large-scale search engines on the web, very little academic research has been done on them. Furthermore, due to rapid advance in technology and web proliferation, creating a web search engine today is very different from three years ago. This paper provides an in-depth description of our large-scale web search engine -- the first such detailed public description we know of to date. Apart from the problems of scaling traditional search techniques to data of this magnitude, there are new technical challenges involved with using the additional information present in hypertext to produce better search results. This paper addresses this question of how to build a practical large-scale system which can exploit the additional information present in hypertext. Also we look at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want.

/pdf/the-anatomy-of-a-large-scale-hypertextual-web-search-engine-496poj789d.pdf

The anatomy of a large-scale hypertextual Web search engine

Imagined communities: Reflections on the origin and spread of nationalism

The Anatomy of a Large-Scale Hypertextual Web Search Engine.

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

The development of the Internet in recent years has made it possible and useful to access many different information systems anywhere in the world to obtain information. While there is much research on the integration of heterogeneous information systems, most commercial systems stop short of the actual integration of available data. Data fusion is the process of fusing multiple records representing the same real-world object into a single, consistent, and clean representation.This article places data fusion into the greater context of data integration, precisely defines the goals of data fusion, namely, complete, concise, and consistent data, and highlights the challenges of data fusion, namely, uncertain and conflicting data values. We give an overview and classification of different ways of fusing data and present several techniques based on standard and advanced operators of the relational algebra and SQL. Finally, the article features a comprehensive survey of data integration systems from academia and industry, showing if and how data fusion is performed in each.

Data fusion

The recent advances in web 2.0 technologies and the rapid adoption of smart phones raises many opportunities for public services to improve their services by engaging their users (who are also owners of the service) in co-design: a dialog where users help design the services they use. To investigate this opportunity, we began a service design project investigating how to create repeated information exchanges between riders and a transit agency in order to create a virtual "place" from which the dialog on services could take place. Through interviews with riders, a workshop with a transit agency, and speed dating of design concepts, we have developed a design direction. Specifically, we propose a service that combines vehicle location and "fullness" ratings provided by riders with dynamic route change information from the transit agency as a foundation for a dialog around riders conveying input for continuous service improvement.

http://www.ri.cmu.edu/pub_files/2010/4/CHI_2010_yoo.pdf

Understanding the space for co-design in riders' interactions with a transit service

In models to generate program source code from natural language, representing this code in a tree structure has been a common approach. However, existing methods often fail to generate complex code correctly due to a lack of ability to memorize large and complex structures. We introduce ReCode, a method based on subtree retrieval that makes it possible to explicitly reference existing code examples within a neural code generation model. First, we retrieve sentences that are similar to input sentences using a dynamic-programming-based sentence similarity scoring method. Next, we extract n-grams of action sequences that build the associated abstract syntax tree. Finally, we increase the probability of actions that cause the retrieved n-gram action subtree to be in the predicted code. We show that our approach improves the performance on two code generation tasks by up to +2.6 BLEU.

Retrieval-Based Neural Code Generation

One formidable problem in language technology is the word sense disambiguation (WSD) problem: disambiguating the true sense of a word as it occurs in a sentence (e.g., recognizing whether the word "bank" refers to a river bank or to a financial institution). This paper explores a strategy for harnessing the linguistic abilities of human beings to develop datasets that can be used to train machine learning algorithms for WSD. To create such datasets, we introduce a new interactive system: a fun game designed to produce valuable output by engaging human players in what they perceive to be a cooperative task of guessing the same word as another player. Our system makes a valuable contribution by tackling the knowledge acquisition bottleneck in the WSD problem domain. Rather than using conventional and costly techniques of paying lexicographers to generate training data for machine learning algorithms, we delegate the work to people who are looking to be entertained.

/pdf/word-sense-disambiguation-via-human-computation-5fohqpjobc.pdf

Word sense disambiguation via human computation

Extensive interviews with riders of the Pittsburgh, Pennsylvania, bus system revealed that, as the top priority, riders wanted to know the actual arrival time of buses. Following a universal design approach, a system called Tiramisu was created to foster a greater sense of community between riders and transit bus service providers. The design focused on acquisition of crowdsourced information for bus location and bus fullness. On the basis of that input, the system predicted the arrival time of buses and provided a convenient platform for reporting problems and positive experiences within the transit system. The intention was to create a community of riders that materially participated in the delivery of the transit service. Tiramisu also supported specific information and reporting needs for riders with disabilities and thereby provided greater independent mobility around the community. An early field trial of Tiramisu suggested that the approach was both feasible and potentially viable.

Mobile Transit Information from Universal Design and Crowdsourcing

From beginning of to the end of the IRO DB ESPRIT project has developed tools for accessing relational and object oriented databases in an integrated way The sys tem is based on the ODMG standard as pivot model and language It consists of three lay ers The local layer provides for an ODMG in terface to heterogeneous DBMSs the commu nication layer implements object oriented re mote data access and the interoperable layer supports design and querying of integrated views This paper describes the architecture and main design choices of IRO DB and re views them against the experiences gained with implementation and application It con cludes with analyzing the revisions and ex tensions needed for applying the developed technology to inter and intranet federations which are tackled in the follow up ESPRIT project MIRO Web

Anthony Tomasic

Papers

Understanding the space for co-design in riders' interactions with a transit service

Retrieval-Based Neural Code Generation

Word sense disambiguation via human computation

Mobile Transit Information from Universal Design and Crowdsourcing

Experiences in Federated Databases: From IRO-DB to MIRO-Web