In this paper, we present Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext. Google is designed to crawl and index the Web efficiently and produce much more satisfying search results than existing systems. The prototype with a full text and hyperlink database of at least 24 million pages is available at http://google.stanford.edu/. To engineer a search engine is a challenging task. Search engines index tens to hundreds of millions of web pages involving a comparable number of distinct terms. They answer tens of millions of queries every day. Despite the importance of large-scale search engines on the web, very little academic research has been done on them. Furthermore, due to rapid advance in technology and web proliferation, creating a web search engine today is very different from three years ago. This paper provides an in-depth description of our large-scale web search engine -- the first such detailed public description we know of to date. Apart from the problems of scaling traditional search techniques to data of this magnitude, there are new technical challenges involved with using the additional information present in hypertext to produce better search results. This paper addresses this question of how to build a practical large-scale system which can exploit the additional information present in hypertext. Also we look at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want.

/pdf/the-anatomy-of-a-large-scale-hypertextual-web-search-engine-496poj789d.pdf

The anatomy of a large-scale hypertextual Web search engine

Imagined communities: Reflections on the origin and spread of nationalism

The Anatomy of a Large-Scale Hypertextual Web Search Engine.

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

The development of the Internet in recent years has made it possible and useful to access many different information systems anywhere in the world to obtain information. While there is much research on the integration of heterogeneous information systems, most commercial systems stop short of the actual integration of available data. Data fusion is the process of fusing multiple records representing the same real-world object into a single, consistent, and clean representation.This article places data fusion into the greater context of data integration, precisely defines the goals of data fusion, namely, complete, concise, and consistent data, and highlights the challenges of data fusion, namely, uncertain and conflicting data values. We give an overview and classification of different ways of fusing data and present several techniques based on standard and advanced operators of the relational algebra and SQL. Finally, the article features a comprehensive survey of data integration systems from academia and industry, showing if and how data fusion is performed in each.

Data fusion

The web contains many datasets presented visually, whose lack of semantic markup renders them difficult to understand and navigate using a screen reader. In this work, we explore the possibility of understanding the semantics of web datasets by asking sighted web users to manually scrape web pages into spreadsheets. Web users constitute a huge population of potential workers, but most are not programmers and may have difficulty understanding and communicating the abstractions involved in labeling web datasets. We present the design of a tool we call SmartWrap that directs the manual scraping work of everyday end users, explicitly including nonprogrammers, towards the construction of reusable programs, called wrappers, that map the scraped website into a structured dataset. To engage with nontechnical end users, we designed the tool to use very simple interactions. We present a user study validating that users with a variety of technical backgrounds were able to use it to construct wrappers. We also validate the SmartWrap approach to acquiring wrappers for a large part of the web by evaluating wrappers contributed by MTurk users using the tool. From the MTurk work we derive estimates for the costs of eliciting wrappers from the crowdworkers for a large proportion of the sampled datasets, and an estimate of what proportion of web datasets can be wrapped by crowdworkers using the tool. We also report some evidence that web users will contribute dataset semantics to the system without pay, motivated either to improve web accessibility or to expedite their own scraping tasks.

SmartWrap: seeing datasets with the crowd's eyes

Administrators frequently perform data integration “by hand” on the desktop as part of the execution of administrative tasks. This position paper discusses the application of mixed-initiative design to this problem. This design style leverages the interaction between a user and an intelligent assistant, minimizing the effort required to execute a task.

/pdf/user-constructed-data-integration-via-mixed-initiative-3a9vstj8bu.pdf

User constructed data integration via mixed-initiative design

DISCO est un systeme de mediation developpe a l'INRIA pour acceder a des sources de donnees heterogenes reparties sur Internet. Dans DISCO, l'utilisateur pose des requetes au mediateur. Le mediateur traite les requetes en accedant aux donnees via les adaptateurs, et retourne la reponse a l'utilisateur. Pour etre efficace le mediateur optimise les requetes en se basant sur l'estimation de leur cout. L'estimation du cout est difficile car les sources heterogenes n'exportent pas d'information de cout. La solution apportee avec DISCO consiste a combiner un modele de cout generique avec des informations de cout exportees par les adaptateurs pour permettre au mediateur d'estimer le cout des requetes heterogenes. Dans cet article, nous validons le modele de cout de DISCO par une experimentation sur des sources de donnees reelles accessibles sur le Web. Cette validation montre l'efficacite de notre modele de cout generique ainsi que l'efficacite de fonctions de cout plus specialisees.

Validating mediator cost models with DISCO

Participatory sensing systems use people and their smartphones as a sensing infrastructure, and getting people to make contributions remains a critical challenge. Little work details how system designers should combine different interactions to increase coverage of service location. Tiramisu, a participatory sensing system, invites transit riders to crowdsource real-time arrival information by sharing location traces when they commute. We extended this system with a new feature that allows riders at stops to "spot" buses passing by. To better understand the impact of this new feature, we conducted an observational log analysis, examining changes in coverage and user behavior before and after the new feature. Following the addition of the spotting feature, participants' contributions increased coverage (the number of trips with real-time data) by 98%, and they used the app more than twice as much. The addition of the spotting feature was also followed by a significant increase of trace contributions.

Combining contribution interactions to increase coverage in mobile participatory sensing systems

Current database performance optimizations stop at the border between the database application and the database system, focusing either on improving the performance of just the database system or the application’s execution in isolation of the other. We argue that typical database application design enables a more holistic analysis that maintains the relationship between the database and application data. We describe techniques to maintain this relationship and introduce several optimizations to improve the efficiency of Web application execution in a distributed environment. We show that our holistic analysis outperforms traditional nonholistic methods both statically and when used as part of a dynamic, distributed environment for executing Web applications using database caches.

Anthony Tomasic

Papers

SmartWrap: seeing datasets with the crowd's eyes

User constructed data integration via mixed-initiative design

Validating mediator cost models with DISCO

Combining contribution interactions to increase coverage in mobile participatory sensing systems

Holistic Application Analysis for Update-Independence

Anthony Tomasic

Papers

SmartWrap: seeing datasets with the crowd's eyes

User constructed data integration via mixed-initiative design

Validating mediator cost models with DISCO

Combining contribution interactions to increase coverage in mobile participatory sensing systems

Holistic Application Analysis for Update­-Independence

Holistic Application Analysis for Update-Independence