Ontologies are becoming so large in their coverage that no single person or a small group of people can develop them effectively and ontology development becomes a community-based enterprise. In this paper, we discuss requirements for supporting collaborative ontology development and present Collaborative Protege--a tool that supports many of these requirements, such as discussions integrated with ontology-editing process, chats, and annotations of changes and ontology components. We have evaluated Collaborative Protege in the context of ontology development in an ongoing large-scale biomedical project that actively uses ontologies at the VA Palo Alto Healthcare System. Users have found the new tool effective as an environment for carrying out discussions and for recording references for the information sources and design rationale.

https://link.springer.com/content/pdf/10.1007%2F978-3-540-88564-1_2.pdf

Supporting Collaborative Ontology Development in Protégé

Ontology evaluation has proven to be one of the more difficult problems in ontology engineering. Researchers proposed numerous methods to evaluate logical correctness of an ontology, its structure, or coverage of a domain represented by a corpus. However, evaluating whether or not ontology assertions correspond to the real world remains a manual and time-consuming task. In this paper, we explore the feasibility of using microtask crowdsourcing through Amazon Mechanical Turk to evaluate ontologies. Specifically, we look at the task of verifying the subclass--superclass hierarchy in ontologies. We demonstrate that the performance of Amazon Mechanical Turk workers (turkers) on this task is comparable to the performance of undergraduate students in a formal study. We explore the effects of the type of the ontology on the performance of turkers and demonstrate that turkers can achieve accuracy as high as 90% on verifying hierarchy statements form common-sense ontologies such as WordNet. Finally, we compare the performance of turkers to the performance of domain experts on verifying statements from an ontology in the biomedical domain. We report on lessons learned about designing ontology-evaluation experiments on Amazon Mechanical Turk. Our results demonstrate that microtask crowdsourcing can become a scalable and efficient component in ontology-engineering workflows.

Mechanical turk as an ontology engineer?: using microtasks as a component of an ontology-engineering workflow

Today, distributed teams collaboratively create and maintain more and more ontologies. To support this type of ontology development, software engineers are introducing a new generation of tools. However, we know relatively little about how existing large-scale collaborative ontology development works and what user workflows the tools must support. In this paper, we analyze our experience in supporting several such projects. We describe a visual and interactive project-management tool that we have developed, which helps ontology developers explore historical ontology change and discussion data. We present the results of qualitative and quantitative studies of the collaborative activity associated with three large-scale ontology-development projects. Based on the analysis, we conclude that domain and ontology experts have different patterns of ontology editing behavior, which has important implications for ontology-development tools.

An analysis of collaborative patterns in large-scale ontology development projects

A holistic approach to collaborative ontology development based on change management

The World Health Organization is currently developing the 11th revision of the International Classification of Diseases (ICD-11). ICD is the standard diagnostic classification used in health care all over the world. In contrast to previous ICD revisions that did not have a formal representation and were mainly available as printed books, ICD-11 uses OWL for the formal representation of its content. In this paper, we report on our work to support the collaborative development of ICD-11 in WebProtege--a web-based ontology browser and editor. WebProtege integrates collaboration features directly into the editing process. We report on the results of the evaluation that we performed during a two-week meeting with the ICD editors in Geneva. We performed the evaluation in the context of the editors learning to use WebProtege to start the ICD-11 development. Participants in the evaluation were optimistic that collaborative development will work in this context, but have raised a number of critical issues.

Ontology development for the masses: creating ICD-11 in WebProtégé

As knowledge engineering moves to the Semantic Web, ontologies become dynamic products of collaborative development rather than artifacts produced in a closed environment of a single research group. However, the projects differ--sometimes significantly--in the way the community members can contribute, the different roles they play, the mechanisms they use to carry out discussions and to achieve consensus. We are currently developing a flexible mechanism to support a wide range of collaborative workflows in the Protege environment. In this paper, we analyze workflows for several active projects, and describe the properties of these workflows. We discuss an ontology that we developed to represent different aspects of workflows for collaborative ontology development. This ontology is a key component of the customizable workflow support in Protege. We evaluate the coverage and flexibility of this ontology by using it to represent formally two different collaborative workflows described in the literature, Diligent and BiomedGT. This evaluation demonstrates that our workflow ontology is sufficiently flexible to represent these very different workflows.

Abraham Sebastian

Papers

A Generic Ontology for Collaborative Ontology-Development Workflows