Lawrence Berkeley National Laboratory
Recent Work
Title
Apply or Die: On the Role and Assessment of Application Papers in Visualization.
Permalink
https://escholarship.org/uc/item/9bd62631
Journal
IEEE computer graphics and applications, 37(3)
ISSN
0272-1716
Authors
Weber, Gunther H
Carpendale, Sheelagh
Ebert, David
et al.
Publication Date
2017
DOI
10.1109/mcg.2017.51
Peer reviewed
eScholarship.org Powered by the California Digital Library
University of California
!
This is the author’s version submitted to IEEE Computer Graphics and Applications before final editing and layouting. The
definitive version of this paper will appear in IEEE Computer Graphics and Applications Volume 37, Number 3, May/June 2017.
DISCLAIMER: This document was prepared as an account of work sponsored by the United States Government. While this
document is believed to contain correct information, neither the United States Government nor any agency thereof, nor the
Regents of the University of California, nor any of their employees, makes any warranty, express or implied, or assumes any legal
responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or
represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or
service by its trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement,
recommendation, or favoring by the United States Government or any agency thereof, or the Regents of the University of
California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States
Government or any agency thereof or the Regents of the University of California.
Apply!or!Die:!On!the!Role!and!Assessment!of!!
Application!Papers!in!Visualization!
Gunther H. Weber
1
, Sheelagh Carpendale
2
, David Ebert
3
, Brian Fisher
4
, Hans Hagen
5
,
Ben Shneiderman
6
and Anders Ynnerman
7
1
Lawrence Berkeley National Laboratory & University of California, Davis, USA
2
University of Calgary, Canada
3
Purdue University, USA
4
Simon Fraser University & University of British Columbia, Canada
5
Technische Universität Kaiserlautern, Germany
6
University of Maryland, College Park, USA
7
Linköping Universitet, Sweden
In his controversial 2004 paper “On the Death of
Visualization,”
1
Bill Lorensen asked “can it survive without
customers?” Concerns about the need for documented
impact on applications has inspired outreach to application
stakeholders and efforts to increase the presence and
visibility of application-driven research in VIS conferences.
Despite a general agreement that application-oriented
papers are of value—application papers have won several
honorable mentions, e.g., at the 2016 SciVis and VAST
conferences—it remains an open question as to what
criteria should be applied to judge whether an application
paper merits acceptance. This is evidenced by the wide
range of evaluations that this paper type receives in
conference reviews and in the diversity of opinions voiced
in the discussion between reviewers. These discussions
frequently focus on how best to achieve the goals of
relevance to application problems as well as the more
familiar criteria of technical excellence and methodological
rigor that are expected for IEEE venues (conferences and
journals).
The panel “Application Papers: What are they and how
should they be evaluated?” at IEEE VIS 2016 in Baltimore,
Maryland brought together leading researchers in all three
VIS main areas—InfoVis (Carpendale and Shneiderman),
SciVis (Hagen and Ynnerman) and VAST (Ebert and
Fisher). Our goal was to gain a better understanding of
prevalent views in the VIS community and to start a
discussion on how to evaluate application papers more
consistently. This Visualization Viewpoint summarizes
points that arose in this panel to encourage continued
discussion and derives a first set of conclusions from them.
Current'Trends'Favoring'Application'Papers'
Three major trends that favor an increased emphasis on
application papers emerged from the discussion: 1) the need
for bigger teams with an increased emphasis on integration
and infrastructure in order to realize the ambitious visions
outlined in proposals; 2) changing research policies; and 3)
the increasing number of researchers involved in
interdisciplinary research.
Realizing the “grand vision” of overall data
understanding requires significant infrastructure-
building and integration efforts. One of the main drivers
of VIS research is to achieve the grand vision of enabling
data understanding in science, engineering and society.
Technical VIS papers play an important role in
accomplishing this mission by advancing the state of the art
in algorithms and methods. These ultimately serve as
building blocks or “dots” for achieving this grander vision.
Realizing the overall vision also requires significant
infrastructure-building and integration work to “connect the
dots” to build a complete large-scale system (Figure 1).
From this perspective, application papers are success stories
that demonstrate that we are achieving our goal and
accomplishing this grander vision.
Integrating visualization techniques into a system—
including work with application professionals, the
combination and refinement of individual methods and
detail work to create usable systems—requires large teams
with dedicated researchers working at the interface between
visualization and application. These researchers bridge the
gap between fundamental visualization technology research
and the grander vision of data understanding. Application
papers document these efforts, providing valuable
knowledge to peers on how to build these large scale
systems. Furthermore, researchers at the interface between
visualization and application science—who invest an
immense effort in infrastructure-building and integration—
need a clear career path, and in the traditional evaluation
model this requires publications. Application-oriented
papers are important for these researchers to earn
recognition and to prosper in their careers. The success of
these researchers in turn supports the field, demonstrating
that innovation in newly developed methods can be used
and will have an impact on real-world applications.
Current research policies favor application impact and
societal relevance. Many sources of funding for
visualization—e.g., from the National Science Foundation
(NSF)—focus on basic research. However, many
visualization research groups also rely on funding sources
with an application-driven mission. Government
agencies—such as the U.S. Department of Energy (DOE)
and the National Institutes for Health (NIH)—as well as
private foundations—like the Stiftung für Innovation and
the Keck Foundation—have requirements that visualization
can help to accomplish, but do not propose the development
of new visualization techniques as the ultimate goal.
Funding from these sources is increasingly dependent on a
track record of refereed publications that demonstrate a
group’s accomplishments and the utility of visualization in
accomplishing mission goals.
To remain competitive, groups funded by these
institutions must publish papers that focus on the
combination, adaptation and refinement of existing
techniques to address a particular problem. More
importantly, application papers also provide the “success
stories” necessary to show the potential of visualization
across a wide range of application domains and obtain
funding in the first place. If we “shun” application papers
from VIS venues, we rob ourselves and future generations
of young visualization researchers of important funding
opportunities. This trend is likely to be exacerbated as even
funding agencies focusing on basic research are starting to
encourage or even require collaborations with application
scientists and professionals, e.g., applications are one
method to show “Broader Impact” for NSF proposals.
The Emergence of Interdisciplinary Researchers.
Computation is playing an increasingly important role in
many sciences and the data deluge from simulations and
experiments requires increasingly sophisticated
computational analysis methods. As a result, a new
generation of interdisciplinary students and researchers is
emerging: as researchers in many disciplines become better
versed in computational methods and computer science
programs create new interdisciplinary degrees in
computational science and engineering—such as
computational chemistry, computational neuroscience, etc.
Many schools are also creating new interdisciplinary
programs in data science. This trend has grown to the point
that some students even pursue doctoral degrees in two
disciplines. These researchers are looking for appropriate
venues for publishing their research results—which can be
extremely interesting and valuable to our community—and
we now have the choice between welcoming their
contributions or pushing them to other places. The future
growth of the visualization community—whether it will
grow, stagnate or even shrink may depend on this decision.
Figure 1: The visualization table developed at the Norrköping Visualization Center C, Linköping University, Sweden, is based
on a progression of visualization papers over the past decade
2-9
(left hand side), integrated into an application that has impact
both in the medical domain and is used in science communication. The image on the right hand side shows visitors to the
Mediterranean Museum in Stockholm interacting with a combined surface and volumetric visualization of the mummy
Neswaiu. A fuller account of this project has recently been published as a contributed article in the Communications of the
ACM.
10
Papers of this kind provide valuable success stories to the visualization community showing how to combine and tailor
the contributions of many technical VIS papers into a widely used system.
Image&Plane&
Sweeping 2011
Multi-Resolution&
Volume Rendering&
2004,&2006
Multi-touch&
Table&System&
2011
TF&Based&
Semantics
2010
Boundary&Aware&
Reconstruction&2014
Ambient&Occlusion
2008
Historygrams&
2012
Benefits' and' Contributions' of' Application'
Papers'
While the increasing importance of application papers
is in part motivated by necessity—developing research
infrastructure, funding and adjusting to new trends in
research and education—they have an immensely beneficial
influence on visualization. Most importantly, application-
driven work also guides basic visualization research,
ensuring it solves relevant problems, and accelerates the
rate of scientific discovery due to the challenging
complexity of real-world problems. Furthermore, it
encourages evaluating new visualization research based on
real data. In addition, application papers build a tacit
knowledge about problem domains and communicate
successes and failures to peers and funding agencies.
Applications improve the quality of basic research.
Traditional models of research posit a linear model that
separates basic research from applications. This assumes
that knowledge flows from basic "curiosity-driven"
research into applied research. To a certain degree, there is
disdain in academic circles for applied research. This belief
was promoted in Vannevar Bush’s 1945 manifesto on
“Science: The Endless Frontier.”
12
However, numerous
critics have pointed out the flaws in this conceptualization,
pointing out the substantial successes from integrated
application oriented work.
13,14
For example, applied
research labs—such as the Bell Labs and U.S. National
Laboratories—contribute to both basic and applied
research. These applied projects have produced Nobel Prize
winning research. Similarly, the Fraunhofer Gesellschaft in
Germany as well as the automobile and aerospace
industries successfully produce basic research that is driven
by applications.
It is possible to measure the positive impact of close
collaboration between academic and applied researchers. At
a recent ACM conference for Knowledge Discovery and
Data Mining (KDD)—the top ranked conference in data
mining—out of 1036 submissions those papers written in
collaboration between academic and industry researchers
received a statistically significantly higher rating than pure
academic or pure industry papers.
There are many ways in which applications promote
basic research that accelerates the development of new
visualization techniques. First, applications provide
challenging problems for visualization, which often
generate new research questions that then trigger new
directions in basic research (mathematics, computer
science, visualization), such as during the development of
EventFlow (Figure 2). Real-world problems have strict
requirements on solution quality and ensure that a
developed theory or technique is able to handle meaningful
data, providing an alternative form of empirical test of a
basic research hypothesis. The result is a positive feedback
loop where applied and basic research cross-fertilize each
other.
Application professionals also provide real data sets for
the development and evaluation of visualization methods,
and application-oriented papers make this data and the
Figure 2: EventFlow
11
is a novel tool for event analytics, providing efficient means to visualize databases of records containing
time-stamped categorical event sequences. This figure illustrates EventFlow with a healthcare example, showing a small database
of 18 patient records with three point events (Admitted, Diagnosed, Stroke) and two interval events (Drug A, Drug B). The most
common pattern in the central overview is that 7 patients receive Drug A, after which 4 shift to Drug B, while 3 of them have a
stroke. For 8 patients the first event in their record is a stroke. On the right a search for patients who received Drug A followed by
Drug B, find 6 matches. EventFlow exemplifies the integration of basic and applied research: Using a theory-driven development
approach, collaborating with healthcare professionals with real problems, made it possible to develop appropriate control panels
and innovative visual designs that present data in ways that support problem solving.
insights to be gleaned from it available to the larger VIS
community. For example, the data sets used by the IEEE
SciVis contest and VAST Challenge are now often used as
examples when describing and evaluation new algorithms
in technical papers with a positive impact on evaluation
sections of VIS papers. Finally, discussions with
application professionals can be inspiring and illuminating
as they provide a fresh perspective. Seeing how our new
techniques are used to gain new knowledge is also an
extremely rewarding experience.
The types of contributions that an application paper can
make to improving basic research include the following:
C-1 Put previous research contributions in an application
context, and describe a combination of methods to
accomplish a visualization/analysis objective;
C-2 Present valuable application specific contributions,
such as tailoring of existing methods;
C-3 Provide a foundation for future visualization
research and pose research challenges; and
C-4 Provide new means (data sets, evaluation criteria) for
the evaluation of visualization methods.
Building deep knowledge about an application domain
and its tools. A good application paper will go beyond
proposing methods for visualizing data from a given
application domain to demonstrating support for the work
processes, including cognitive work, that experts currently
use. The best application papers go beyond current work
practices to propose new technological capabilities for
domain experts that will enable them to improve on their
current practices. For these projects, it is necessary for
visualization designers to conduct a deep analysis of tacit as
well as explicit knowledge in the domain. Contributions
from these papers include:
C-5 Conveying the “language” of the application domain
and assessment methods for particular situations, thus
facilitating deep collaborations with application
professionals.
C-6 Enabling the VIS community to learn how researchers
in other communities approach problems as individual
analysts and in collaboration with others.
C-7 Documenting successes and failures of visualization
approaches for a given application domain and
deriving lessons learned from the visualization
experts perspective; and
C-8 Discussing both design methodologies and individual
design decisions that have proven to successfully
address the users’ need.
Papers that make contributions at the level of cognitive
and collaborative work practices will have a stronger
impact on how the VIS community can best approach
visualization for a given application domain.
Application papers provide success stories. In addition to
providing success stories for funding agencies, application
papers also communicate the successful application of VIS
to many different fields. These success stories are useful for
VIS researchers to see what methods work and what
methods need improvement as well as see how their
techniques and algorithms are applied to real-world data.
The latter is helpful, for example, in steering future work on
VIS techniques. Finally, success stories help in attracting
new collaboration partners and create new ideas for VIS
techniques.
Contributions in this area include:
C-9 Creating awareness in application domains of the
importance of visualization;
C-10 Documenting a success story for dissemination,
both inside and outside of the visualization
community;
C-11 Describing innovation processes and commercial
impact; and
C-12 Contributing to a wide-spread understanding of
visualization science.
Application papers do not fit a set “template.” An
application paper can thus serve many different purposes
and it is therefore not possible to describe one “template”
for how to document these different contributions. Quite
the contrary, several panel participants argued that “one
size doesn’t fit all” and that there should be no fixed rules
to write such a paper. Shoehorning an application paper
into rigid guidelines could prevent the paper “singing the
way it should.” There was consensus however, that a paper
should make contributions to our own VIS community and
not just to the application domain, i.e., that an application
paper should go beyond an “instruction manual.” While the
paper should be targeted at an audience whose core
competencies consist of designing new visualization and
analysis methods and evaluating them, there are many ways
in which an application paper can contribute to the state of
the art of VIS research.
Assessment'in'the'Review'Process'
The variance of application papers and the “one size
doesn’t fit” all makes application papers difficult to review
and assess. This does not mean that they should not be
evaluated, but rather that reviewers should consider the
whole spectrum of possible contributions, some of which
are mentioned in the previous section. It would, however,
be an interesting exercise to look at each of these
contributions and see how they would best be described and
put forward in guidelines for authors.
Weighing technical and other contributions. Application
papers contribute in many different areas, and reviewers
should not expect an application paper to present a new
