Fluid Interaction for Information Visualization
Niklas Elmqvist
∗
Andrew Vande Moere Hans-Christian Jetter Daniel Cernea
Harald Reiterer T. J. Jankun-Kelly
Abstract
Despite typically receiving little emphasis in visualization research, interaction in visualization is the catalyst for the user’s
dialogue with the data, and, ultimately, the user’s actual understanding and insight into this data. There are many possible
reasons for this skewed balance between the visual and interactive aspects of a visualization. One reason is that interaction
is an intangible concept that is difficult to design, quantify, and evaluate. Unlike for visual design, there are few examples
that show visualization practitioners and researchers how to best design the interaction for a new visualization. In this paper,
we attempt to address this issue by collecting examples of visualizations with “best-in-class” interaction and using them to
extract practical design guidelines for future designers and researchers. We call this concept fluid interaction, and we propose
a operational definition in terms of the direct manipulation and embodied interaction paradigms, the psychological concept of
“flow”, and Norman’s gulfs of execution and evaluation.
Keywords: fluidity, flow, embodiment, design, information visualization, human-computer interaction.
1 Introduction
Not all information visualization (InfoVis) tools are created equal. As all users of such tools know, while we can expect any
competent tool to be able to represent abstract data in graphical form, there is a certain class of InfoVis tools that take this
a step further through engaging, compelling, and even absorbing user experiences that turn the analytical sensemaking [45]
process into a pleasurable task. However, the academic portion of the InfoVis field in general puts little emphasis on design,
aesthetic, and user experience aspects of information visualization tools, and so far there has been virtually no effort towards
characterizing this class of InfoVis tool in the research community. In contrast, much of information visualization in the real
world is directly concerned with creating compelling—even playful—and beautiful tools that are capable of capturing the
attention of general users on the Internet and in public spaces such as museums, exhibition halls, and corporate lobbies.
In this article, we try to remedy this state of affairs by proposing a unifying concept for both researchers and practitioners
that captures this class of InfoVis exemplars in a single definition: fluid interaction. Fluidity in information visualization is
an elusive and intangible concept characterized by smooth, seamless, and powerful interaction; responsive, interactive and
rapidly-updated graphics; and careful, conscientious, and comprehensive user experiences. Our hypothesis is that an InfoVis
tool that exhibits this fluidity in all aspects will transform the sensemaking process into an efficient, illuminating, and even
enjoyable experience because it helps the user stay in the flow [13] of the work process. However, creating a fluid design is far
from trivial, mainly because of this intangibility and elusiveness.
To better illustrate our definition of fluid interaction for information visualization, we collect and describe a subset of
these exemplar InfoVis tools from the scientific community, including BabyNameVoyager [58], Facet-Streams [30], and Scat-
ter/GraphDice [9, 17]. Unlike most existing research articles in information visualization, we also collect exemplars from
outside the scientific community, such as from the design, aesthetics, and infographics communities. These “real-world”
exemplars include mæve [4, 36, 39], We Feel Fine [22], and the interactive holographics from the film Iron Man 2 [12].
Using these exemplars as a starting point, we derive tips, guidelines, and principles for how to achieve fluidity in InfoVis in
terms of both interaction and visual representations. We hope that these snippets of practical experience, elevated almost to the
level of being design idioms and patterns [21], will help developers of InfoVis tools—academics and practitioners alike—to
build better, more rewarding, more captivating, and ultimately more efficient information visualization tools that will propel
our field to make an even bigger impact in both our own scientific community as well as the real world.
In the rest of this paper, we first survey the state of the art in fluid interaction, design, and user experience for information
visualization (Section 2). We then attempt to define the concept of fluid interaction through a detailed list of properties that we
think characterize fluid InfoVis tools (Section 3). These properties are all exhibited in the context of our InfoVis exemplars, that
∗
School of Electrical & Computer Engineering, Purdue University, 465 Northwestern Ave, West Lafayette, IN 47907, USA, E-mail: elm@purdue.edu.
1
we describe next (Section 4). Starting from these exemplars, we derive design guidelines for fluid information visualization
(Section 5), and conclude the paper with our vision for future research directions in design aspects of information visualization
(Section 6).
2 Related Work
Unlike its sister field of human-computer interaction, where design and user experience are major components, InfoVis research
articles generally do not place much emphasis on interaction design aspects of information visualization. The few papers that
discuss these topics tend to do so from a purely scientific, engineering, or implementation viewpoint. For example, although
Amar et al. [2] present a meta-level review of typical analysis t+asks and Yi et al. [59] collect seven archetypes of interaction,
these are all still descriptive classifications, which makes them difficult to use in a generative and design purpose. They also
do not describe “softer” and less goal-driven aspects of interaction design such as aesthetics, user experience, and rewarding
interaction.
Below we discuss both the visual and the interactive aspects of visualization design.
2.1 Visual Aspects
Textbooks on graphic design are an excellent source of information on maximizing clarity and expressiveness in static (often
printed) visual representations. Bertin [8] discuss retinal variables for visual marks in data visualization (later improved and
extended by Mackinlay [35] and again by Card et al. [11]). Furthermore, Edward Tufte’s books [50, 51, 52, 53] are key
resources for visual design of data displays, and have influenced many InfoVis systems and papers through the years. Following
this tradition, Stephen Few’s books give clear and concrete design guidelines for how to design graphs and tables [18] as well
as information dashboards [19].
Visualization textbooks are typically even more relevant because they focus on interactive visualization applications. Colin
Ware’s books [56, 57] on perception and cognitive aspects of visualization provide valuable background on the psychology of
visual thinking for interactive visualizations. Robert Spence takes a design and interaction-oriented approach to information
visualization in his textbook [48]. A recent book, Beautiful Data [46], gives a hands-on approach to visualization design
through a set of case studies involving real datasets collected from the web. Most recently, Ward et al. [55] devote a full chapter
to step-by-step guidelines on how to design effective visualizations.
2.2 Interactive Aspects
While visual aspects are important for fluid interaction, it is clearly the interactive aspects that are central for effective visual-
ization interaction design. Unfortunately, interaction is not discussed at all in graphic design, and even visualization textbooks
tend to downplay this angle.
One exception is Robert Spence’s book [48], which takes interaction design as its starting point for the study of information
visualization. Many of the visualization techniques presented in the book include a discussion on key interaction design aspects
associated with the visual representation. Another exception is Stephen Few’s newest book [20], which incorporates a chapter
on analytical interaction where Few gives recommendations for how visualization software should best support the analytical
discourse. Few even uses the word “fluid” to describe a desirable feature of the interactive exploration process ( [20], pp. 82).
Our work in this paper builds on these existing efforts, but we formalize the concept of flow and fluidity further.
The recent paper by Pike et al. [42] on the “science of interaction” is highly relevant to our work. Pike makes a case for the
role of interaction in visualization and visual analytics, and emphasizes themes such as interaction design, user experiences,
and best practices for interactive tools—the very same themes we promote here. The paper ends with seven broad areas for
future research, but does not take the practical design approach that our paper does.
Given the dearth of interaction design in InfoVis research, it is fortunate that there exists much HCI research that is highly
relevant to the concepts of fluidity and flow in visualization. We draw on a wide variety of disciplines—HCI and interaction
design included—when we define fluid interaction in the next section and we will thus cite and discuss these sources below.
However, the most influential work includes the direct manipulation paradigm [47], Norman’s gulfs of execution and eval-
uation [41], the instrumental interaction model [5, 6], tangible [28] and embodied interaction [15], and, most recently, the
reality-based interaction framework [29].
In the next section, we show how this rich collection of related work, theories, and frameworks can be tied together into the
concept of fluid interaction and how it can be applied to information visualization.
2
3 Fluid Interaction for Information Visualization
As can be seen from the above literature survey, there exists very little work on the topic of fluid interaction for information
visualization. Part of the reason for this is that notions of flow [13] and fluidity are very elusive and difficult to pin down [7].
In this section, we will make inroads towards an operational definition of this concept. In the next section, we will describe a
set of existing applications that fulfill this definition: we call them InfoVis exemplars. Finally, in the section following that, we
synthesize the characteristic properties collected from these examplars into general guidelines for how to design fluid InfoVis
applications.
3.1 Operational Definition
As mentioned above, fluid interaction is an elusive concept that is not easily amenable to a theoretical definition. Instead,
we present an operational definition of the properties of fluidity that draws from a plethora of sources, including HCI design,
the concept of flow [13], embodiment [15], immersion, and natural interaction. As a starting point, Merriam-Webster defines
fluidity as follows:
fluidity, n.: 1. the quality or state of being fluid; 2. the physical property of a substance that enables it to flow.
Following this general definition, a fluid interface for information visualization is characterized by one or several of the
following properties:
• Promotes flow: The interaction should be designed to promote staying in the flow. “Flow” [13] is defined as a mental
state of total immersion in an activity where the challenges of the activity and the skills of the participant are perfectly
balanced, leading to high focus, involvement, and rewarding outcomes. Bederson [7] previously proposed an interaction
design philosophy based on helping the user stay in the flow, emphasizing five characteristics for user interface design.
In the below list, we take a somewhat broader view of Csikszentmihalyi’s factors for flow that may influence the concept
of fluid interaction:
– Balanced challenge: the skill required by the activity and the user’s skill level should be matched;
– Concentration: the activity should allow for a high degree of focus on a limited field of attention;
– Loss of self-consciousness: the user should be able to merge action and awareness;
– Transformation of time: enable users to “lose themselves” in the activity, essentially losing track of time;
– Prompt feedback: users should be immediately informed of progress towards their goals;
– Sense of control: ensure users feel in control over the activity so that they can truly affect the outcome; and
– Intrinsically rewarding: the activity should have a tangible reward in and of itself.
• Supports direct manipulation: The direct manipulation paradigm [47] (further extended by the instrumental interaction
model [5, 6]) promotes an explicit method of interacting with computers by directly interacting with the domain objects
themselves, thereby minimizing the indirection in the interface. The paradigm is based on four main principles:
– Continuous representation of the object of interest;
– Physical actions instead of complex syntax;
– Rapid, incremental, and reversible operations whose impact on the object of interest is immediately visible; and
– Layered or spiral approach to learning that permits usage with minimal knowledge.
• Minimizes the gulfs of action: According to usability expert Donald Norman, the challenge of interacting with any
system, physical or virtual, can be described in terms of two gulfs [41]:
– Gulf of Evaluation: The difference between the system’s state and the user’s perception of that state.
– Gulf of Execution: The difference between the allowable actions of a system and the user’s intentions for using the
system.
3
3.2 Towards a Cognitive Account of Fluid Interaction
From a cognitive perspective, the aforementioned desired properties of a fluid interface for InfoVis share some important com-
monalities. Understanding these commonalities and identifying an underlying common concept could help to better understand
what creates and what hampers fluidity in InfoVis. In the following, we suggest such a common concept which is deduced
from existing cognitive models. This first step towards a cognitive account of fluidity is still far from being a comprehensive
theoretical model. However, viewing visualization design through this new lens could help to analyze existing information
visualizations and to inform and refine future models.
In our view, a basic requirement for fluidity and the underlying concept behind it is the users’ feeling of direct participation
and embodiment in the interface. Fluid interfaces for InfoVis must make the users feel that are able to directly “touch” and
manipulate the visualization instead of indirectly conversing with a user interface. Users should get a feeling of immersion,
first-personness and direct engagement with the objects and the visualizations that concern them.
This phenomena was first described by Hutchins et al. [27] in their cognitive account of direct manipulation. They differen-
tiated between two major metaphors for the nature of human-computer interaction, a conversation metaphor and a model-world
metaphor: in the former, the interface serves as a language for interacting with the world, whereas in the latter, the interface
itself is the world which the user can manipulate. For Hutchins et al., model-world interfaces create a feeling of directness
and direct manipulation by minimizing the gulfs of evaluation and execution [41] and thereby using much less of the users’
cognitive resources.
We believe, that InfoVis—and in particular fluid InfoVis—should follow this model-world metaphor. This is also in line
with the growing importance of theories of embodied cognition in cognitive science and embodied interaction in human-
computer interaction [15]. These embodied views emphasize that our cognitive abilities are specifically designed to reason,
act, and move in our natural physical and social world. In other words, our entire way of thinking and our perception are
optimized for these real-world tasks and cannot be separated from our physical and social existence. These physical and social
skills are far more constituent of what we consider as human cognition than the disembodied and formal processing of symbols
when conversing with a computer by clicking through labelled buttons, menus, hyperlinks or forms. Thus, we are wasting a
great deal of our true skills when using computing technology without directly acting in metaphorical (model-)worlds.
This insight has lead the field of HCI to a new generation of user interfaces based on reality-based interaction [29]. These
interfaces use modalities such as body tracking, bi-manual multi-touch interaction, or tangible objects to further reduce the
users’ gulf of execution and to draw strength by employing themes of reality such as body, environment and social skills &
awareness [29]. Thus they transfer an even stronger form of direct manipulation from desktop to post-desktop computing.
In conclusion, given our innate cognitive abilities, creating model-world interfaces into which users can immerse them-
selves and in which objects of concern become virtually or even physically tangible can help to achieve Csikszentmihalyi’s
factors of flow. Directly manipulating the objects in the model-world with a greater set of motor skills (e.g., directly dragging
an object with multi-touch or mouse instead of pushing keys or buttons to that effect) mediates a sense of control and the
model-worlds provide the desired prompt feedback. They use less cognitive resources and thus enable concentration on the
task instead of concentration on handling the user interface. Less usage of cognitive resources can also help to achieve a greater
design space for a more balanced challenge. Furthermore, good direct manipulation interfaces are in many respects similar to
computer games [47], and thus could lead to the loss of self-consciousness and transformation of time.
3.3 Utility of Fluid Interaction
Our argument so far has assumed that fluid interaction is a desirable attribute in visualization design. However, it is true
that fluidity is not a necessary condition for any given visualization application, and we can even go so far as to say that
there probably exist numerous very successful visualization and analysis applications that score low on what we would call
“core” fluidity properties. For example, the statistical package R (http://wwwr-project.org/) has a command-based
user interface with few visual components (i.e., a conversation interface), but is widespread and highly successful in many
communities (we should note that R certainly does promote flow for expert users, while other fluidity properties are somewhat
neglected).
On the other hand, interaction is the catalyst for the interplay between the data and the user, and is an essential part of
visual exploration [14, 40, 59]. Thus it follows that achieving the “optimal experience” (characterized as flow [13]) while
interacting with a visualization application will cause the user to perform better [7]. In other words, if we improve the fluidity
of a visualization application, user performance would directly benefit. This is also the message of this paper: showing how
interaction design can be used to make visualization applications, existing and novel ones alike, more effective. For example,
RStudio (http://www.rstudio.org/) is a new graphical IDE for R that likely will help novice users overcome the steep
learning curve of the R system.
4
Many authors argue in favor of streamlining the interactive process in this way. Pike et al. [42] emphasize the need for
natural and seamless interaction methods in support of discovery. Although they note that disruptions in the analytical discourse
are inevitable and, in fact, often useful, their use of the word “fluid” is different than ours, and their argument still seems to
advocate the user staying in the flow of a visualization tool during analysis. Bederson [7] use interruptions as a reasoning
tool when arguing for maintaining flow in interactive applications, noting that literal or conceptual interruptions can have large
impact on user productivity. While sensemaking literature [45] typically discusses interaction with information at a conceptual
level, many of Pirolli and Card’s leverage points [43] directly or indirectly involve improving the cost structure of analysis
through tool innovation.
Given the above operational definition, cognitive account, and utility for fluidity, we are now ready to study concrete
examples that embody these concepts.
4 InfoVis Exemplars
In this section, we review six InfoVis exemplars that we feel exhibit the fluid interaction properties discussed earlier in this
article. Table 1 gives a summary of the exemplars based on domain (the primary community the system targets), audience
(intended users), task (the primary task and intention with the system), and properties (the primary properties of fluidity that
the system exhibits). We describe each of these systems in detail below.
Of course, any choice of a mere six examples from the rich array of excellent visualization systems is highly subjective and
could even be seen as somewhat arbitrary. We used the following criteria when identifying the exemplars:
• Diversity: We wanted our selection to reflect the practice of information visualization in both academic and design
communities, as well as on the web, in movies, and in physical locations such as museums.
• Illustrative: Our choice was influenced by a desire to illustrate different—seemingly disparate—aspects of each system
that the concept of fluidity could help explain.
• Availability: We chose only work that that was directly available (through demonstrations or videos) and amply de-
scribed in the literature (for academic work) or in the blogosphere or popular press (for non-academic work). In other
words, we relied on peer-review (academic work) or public acceptance (non-academic work).
• Depth instead of breadth: This paper is not a comprehensive survey of best-in-class visualizations, and thus our focus
was on deep analysis of each exemplar instead of a large and broad enumeration of all InfoVis systems that incorporate
fluid interaction.
Other factors beyond our control naturally also played a role in this selection process, including the authors’ own biases,
personal tastes, and familiarity with the literature. However, regardless of the sparse sampling of the design space, we are con-
vinced that the characteristic properties that we extract from each exemplar together are broadly applicable for fluid interaction
in general.
Exemplar Domain Audience Task Properties
Facet-Streams [30] academic groups collaborative search direct, embodied, minimizes gulfs
BabyNameVoyager [58] academic/design web exploration direct, prompt feedback, rewarding, control
Scatter/GraphDice [9, 17] academic analyst exploration direct, prompt feedback, minimizes gulfs
Mæve [36, 39] museum casual browsing embodied, rewarding, aesthetic
We Feel Fine [31] design web social navigation rewarding, aesthetic, minimal knowledge
Iron Man 2 [12] movie casual immersion embodied, feedback, aesthetic
Table 1: Overview of the six InfoVis exemplars reviewed in this paper with informal classifications on the domain, intended
audience, main task, and primary fluid properties of each system.
4.1 Facet-Streams
Facet-Streams [30] is a system for co-located collaborative product search by multiple users around a tabletop (Figure 1). It
supports small groups during decision-making and negotiation by enabling a faceted exploration of a product catalog, e.g. a
catalog of hotels for a family’s vacation. Queries can be formulated by each participant by putting small glass discs as query
tokens on the tabletop and assigning the desired data field and value ranges to them (Figure 2). These tokens can then be
5
![Figure 4: The GraphDice [9] multivariate graph visualization tool showing the current plot (right), the plot matrix (top center) and the query layer box (bottom center). The integrated spreadsheet-style table (left) shows details on demand. A FaSTslider [37] (active in the current plot) allows for quick navigation between data dimensions.](/figures/figure-4-the-graphdice-9-multivariate-graph-visualization-2evdhxhx.webp)