HAL Id: hal-02197062
https://hal.archives-ouvertes.fr/hal-02197062
Submitted on 30 Jul 2019
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
What is Interaction for Data Visualization?
Evanthia Dimara, Charles Perin
To cite this version:
Evanthia Dimara, Charles Perin. What is Interaction for Data Visualization?. IEEE Transactions on
Visualization and Computer Graphics, Institute of Electrical and Electronics Engineers, 2020, 26 (1),
pp.119 - 129. �10.1109/TVCG.2019.2934283�. �hal-02197062�
What is Interaction for Data Visualization?
Evanthia Dimara and Charles Perin
∗
Abstract
—Interaction is fundamental to data visualization, but what “interaction” means in the context of visualization is ambiguous
and confusing. We argue that this confusion is due to a lack of consensual definition. To tackle this problem, we start by synthesizing
an inclusive view of interaction in the visualization community – including insights from information visualization, visual analytics and
scientific visualization, as well as the input of both senior and junior visualization researchers. Once this view takes shape, we look at
how interaction is defined in the field of human-computer interaction (HCI). By extracting commonalities and differences between the
views of interaction in visualization and in HCI, we synthesize a definition of interaction for visualization. Our definition is meant to be
a thinking tool and inspire novel and bolder interaction design practices. We hope that by better understanding what interaction in
visualization is and what it can be, we will enrich the quality of interaction in visualization systems and empower those who use them.
Index Terms—interaction, visualization, data, definition, human-computer interaction.
1 INTRODUCTION
The notion of interaction has been a challenging concept to define in
the field of Human-Computer Interaction (HCI). Only recently, an HCI
review entitled “What is interaction?” [56] summarized concepts that
describe the causal relationships between the human and the computer.
While this HCI review outlines opportunities for enriching interactivity
with computer systems, it is unclear how these concepts relate to visu-
alization. Therefore, while visualization researchers and practitioners
may be aware of such interaction concepts, they do not necessarily see
how to apply them to their own data-oriented practices and needs.
Meanwhile in the past decade we have witnessed a growing call
for enriching interactivity in visualization systems. Forward-looking
research on interaction for visualization advocate for visualization
systems that give absolute freedom to end users to actively restruc-
ture [73, 100], sketch [75], author [69, 119] and personalize [57, 111]
visualizations; to construct visualizations from scratch [59], perform
data-aware annotations on them [54, 117], and unruly remove distract-
ing information [29]; to enrich visualizations with external knowl-
edge [118], control fluently [36] both data presentations [101] and data
pre-processing statistical functions [37]; to indicate uncertainty [82],
collaborate with peers [55, 80], and interact with visualizations using
natural means [66, 74] within physically situated settings [63, 73, 121].
We argue that a strong barrier to achieving this vision is not only that
of the technical challenges, but like in HCI, that of defining interaction
for visualization. In the visualization pipeline [16], interaction occurs at
all stages of the visualization process of turning raw data into views on
the data. While the visualization community has iteratively structured
and formalized the representation aspect of the pipeline, significantly
less attention has been paid to the interaction aspect [36, 72, 74, 99].
The nature and role of interaction has actually sparked discussions and
arguments since the visualization field was created. As of today there is
no consensus on what interaction is, and what its role for visualization
is – as interaction is an elusive and overloaded term [74, 92, 126].
To address this problem, we first capture the current view of interac-
tion from the visualization community based on the input of researchers.
Once this view takes shape, we revisit the view of interaction from the
HCI community [56] to understand how the two views differ and relate
to each other. Combining these two perspectives, we then propose a
• Evanthia Dimara is with Sorbonne University. E-mail:
evanthia.dimara@gmail.com.
• Charles Perin is with University of Victoria. E-mail: cperin@uvic.ca.
∗
All authors contributed equally to this manuscript, with the exception of the
first author who did most of the work.
This is the author’s version of the work.
To appear in IEEE Transactions on Visualization and Computer Graphics.
definition of interaction for visualization. This definition attempts to
broaden the scope of interaction in visualization and is inclusive as it
considers the perspectives of information visualization, visual analyt-
ics, and scientific visualization. We further extend this definition to
operationalize flexibility within visualization systems, based on where
interactions occur semantically in the visualization pipeline. We hope
that this definition will spur novel, bolder interaction design practices
in visualization, and the growth of more flexible visualization tools.
2 A REVIEW OF INTERACTION FOR VISUALIZATION
To capture the view of interaction in visualization, we started with a list
of papers based on our own expertise and expert input. Then we applied
a recursive process to expand our review while accounting for our own
bias. We describe our methodology rationale, method for collecting
papers, questionnaire we sent to experts, paper collection method, and
tagging method. Then we present the summary statistics of the review.
2.1 Methodology Rationale
The topic of interaction in visualization is broad; arguably most visual-
ization papers mention interaction at a point. Thus we discarded the
systematic review and instead opted for a critical review, that needs not
be exhaustive but requires a more detailed examination of the litera-
ture [46]. Because critical reviews seek to identify the most significant
items in the field [46], we set the three following requirements:
R1:
: The view of interaction of the community cannot be captured only
by citation number, it needs to include expert input.
R2:
: The snowballing approach alone (starting with a set of seed papers
and expanding using back- and forward-references) is not sufficient
because it is biased by the selection of seed papers.
R3:
: The review must not include HCI papers unrelated to visualization.
We used two metrics to measure the impact and relevance of a paper
P based on publication year P
y
. I
I
Im
m
mp
p
pa
a
ac
c
ct
t
t
P
P
P
= (P
c
/10)/(current year −
P
y
+ 1)
measures the importance of the paper to the community based
on its number of citations
P
c
.
R
R
Re
e
el
l
le
e
ev
v
va
a
an
n
nc
c
ce
e
e
P
P
P
= P
f
/(current year − P
y
+
1) + (P
b
+ 1)/(P
y
− 1980 + 1)
measures the relevance to the topic of
interaction based on its number of forward references
P
f
and backward
references
P
b
(i.e., papers that cite, and that are cited by,
P
) that contain
both “interaction” and “visualization” in their title. 1980 is the year of
the oldest paper [17] we collected. These metrics identify both highly
relevant and impactful papers, while not relying solely on bibliometrics.
We set the inclusion criteria to
Impact
P
> 0.5
, keeping papers with
roughly more than 10 citations a year, and to
Relevance
P
> 0.2
, keeping
papers with roughly more than two forward or backward references
with the keywords “interaction” and “visualization” in their title every
10 year. These cutoffs (which are broad to prevent false negatives)
include important papers but exclude some clearly non-relevant ones.
2.2 Review Questionnaire
We sent visualization researchers an online form asking for: (1) defini-
tion papers, the papers that attempt to define interaction in visualization;
(2) relevant papers to the topic of interaction; (3) their years of visu-
alization experience; and (4) their interaction experience on a 7-point
scale describing how often their own papers focus on interaction. There
were also three optional fields: name, email, and comments.
2.3 Method for Collecting Papers
We created a list of seed papers with the following snowballing method:
Step 1: We started from our own list of 5 definition papers (R1).
Step 2:
We sent the questionnaire to expert visualization researchers
who suggested both definition and linked papers (R1,R2).
Step 3: We included in the list of seed papers each paper P that:
•
was included in our initial list of 5 definition papers or was
suggested as a definition paper at least once; and
•
is a journal article or conference paper – to avoid non-peer
reviewed entries such as demos; and
• has the term “visualization” in the title or abstract (R3); and
• has Relevance
P
> 0.2 and Impact
P
> 0.5.
In addition, considering that expert input can provide insights not
captured by our computational method (
R1
), we included in the
list of seed papers those that did not fulfill these inclusion criteria
but that were suggested as definition paper three times or more.
Step 4:
We repeated Step 2, sending the questionnaire to all (non-
previously contacted) authors of the added seed papers. We
then repeated Step 3 with the new suggestions. We stopped the
recursion when no new seed paper or author was found.
Once the list of seed papers finalized, we derived a set of linked papers.
A paper P is linked if it is not seed and one of the following is true:
• P is suggested by two or more experts as definition or relevant;
• P
cites or is cited by a seed paper and its title contains both
“visualization” and “interaction”.
2.4 Method for Tagging Papers with Interaction Themes
We (the two authors of this paper) developed tags (short labels) to
characterize the seed and linked papers using an iterative deductive
coding method [88]. Specifically, we identified emerging themes from
reading seed papers with the lens of capturing: i) the current view of
interaction in visualization; ii) problems raised by the community; and
iii) existing archetypal descriptions and categorizations of interactions.
We then refined these tags incrementally until we obtained a high
inter-coder agreement. We then coded all seed and linked papers. We
measured inter-coder agreement using Cohen’s Kappa [24] (
K
), that
accounts for the possibility that agreement occurs by chance. There is
no strict rule regarding how to interpret values of
K
, but inter-reliability
is considered satisfactory for K ≥ 0.7 and excellent for K > 0.75 [41].
For the first coding iteration, we used plain tag names. We indepen-
dently tagged 6 seed papers using the tag names described below with
binary values (a paper either has the tag or not). We obtained
K = 0.57
.
We discussed similarities and differences in our tagging, then wrote the
detailed definitions for each tag presented below. We switched to three
values: 0 (not applicable tag), 1 (partial fit of the tag), and 2 (good fit
of the tag). For the second iteration, we independently coded again
the 6 first seed papers (
K = 0.83
), as well as 6 additional seed papers
(
K = 0.78
), showing excellent agreement
(K > 0.75)
. For the third
iteration, we independently coded 12 new seed papers. We obtained
K = 0.87
, which allowed us to have a single coder per remaining paper.
Tag — Definition of Interaction
: attempt to define, explain or de-
scribe interaction. A rating of 1 marks implicit attempts, including
definitions based on specific properties or attributes of interaction. A
rating of 2 marks direct definitions (formal or informal, e.g., “interac-
tion is ...”).
Tag — Critical on Interaction
: complaints and frustrations (using a
negative tone) about interaction, to reveal areas of improvement and
challenges. A rating of 1 marks brief complaints. A rating of 2 marks
extensive discussions of complaints.
Tag — Benefits of Interaction
: positive aspects of interaction (actual
or expected). A rating of 1 marks brief (or moderately phrased) dis-
cussions of benefits. A rating of 2 marks more elaborated (or strongly
phrased) discussions of benefits.
Tag — Interaction Concepts
: archetypal description of interaction.
Given to papers that contribute one of: “concept”, “model”, “frame-
work”, “design space”, “paradigm”. A rating of 1 marks concepts with
small or unclear relation to interaction. A rating of 2 marks concepts
with interaction as a key element (e.g., an interaction model).
Tag — Interactive Pipeline
: discussion of interaction in relation to
the visualization pipeline, a fundamental concept that transcends sub-
domains of visualization and explicitly includes interaction. It is given
to papers that use the word “pipeline” or cite one of [16, 20, 21, 48]. A
rating of 1 indicates that the existence of interaction is unclear. A rating
of 2 indicates that the interaction component is clearly discussed.
Tag — List
: contribution of a list (e.g., of interaction techniques, tasks,
or intents). We call formal a list that is described with one of the follow-
ing words: “taxonomy”, “classification”, “typology”, “categorization”.
A rating of 1 marks an informal review (it does not contain any of the
formal keywords). A rating of 2 marks a formal review.
2.5 Summary Statistics of the Review
1
RECRUITING: We emailed 77 visualization experts over a two-month
period. 64/77 were successfully delivered. 34/77 were in our initial
list of experts and the remaining 43/77 were derived through our re-
cursive algorithm. While we started with a bias toward information
visualization (e.g., [36, 72, 79,106]), our algorithm expanded the scope
naturally to visual analytics (e.g., [37, 45, 50, 92]), scientific visualiza-
tion (e.g., [1, 66, 67, 77]) and digital cartography (e.g., [98, 99]).
PARTICIPANTS: 22 visualization researchers responded to the survey.
They had 5–32 (
M : 13.6
,
SD : 6.2
) years of visualization experience.
12/22 were authors of seed papers, and 14/22 were authors of linked
papers; 2/22 completed the survey anonymously. Participants rated
their interaction expertise with a mean of 5.9/7 (SD : 0.8).
SEED PAPERS: Fig. 1 presents all 59 tagged papers. We started with
5 definition paper based on our expertise. Participants suggested 114
(83 unique) definition papers (6 did not suggest any definition paper).
Applying our recursive algorithm resulted in a set of 23 seed papers.
LINKED PAPERS: Participants suggested 61 (51 unique) relevant papers.
Forward and backward searches gave 105 papers citing, and 104 papers
being cited by a seed paper. Our linked paper collection algorithm
gave 47 linked papers. We excluded 7 theses, 3 unpublished and 1
non-English reports. This resulted in 36 linked papers.
TAGS: The derived tags were: definition: 32; benefit: 39; critical: 36;
concept: 20; pipeline: 12; and list: 35.
3 THE VISUALIZATION VIEW OF INTERACTION
In this section, we synthesize the current view of interaction in visu-
alization based only on the points of view of the 59 seed and linked
papers we reviewed (see Fig. 1). We elaborate on our own critique of
this literature in Sect. 4 and Sect. 5. While we reviewed all 59 papers,
due to space limitations, we only cite papers which have been cited
more than twice – and provide all papers in supplementary material.
3.1 Definitions of Interaction for Visualization
Interaction has been identified as an overloaded [92] and elusive [36,74]
term, and it is challenging to find a solid definition of interaction
[36, 74, 126]. Therefore, we consider the 32 papers with a definition
tag – whether they attempt to define interaction implicitly or explicitly.
3.1.1 What Interaction for Visualization Must Involve
The definition papers reveal mandatory components of interaction,
namely: external and internal entities, external and internal actions.
EXTERNAL ENTITIES: The two most cited mandatory entities are
the user and the data. The user (or analyst [8, 37, 112]) is a human
1
Material: https://osf.io/ej7xg/?view only=51485163dfc94d0c8499af17cb2038b2
THE INFORMATION VISUALIZER, AN INFORMATION WORKSPACE
THE EYES HAVE IT: A TASK BY DATA TYPE TAXONOMY FOR INFORMATION VISUALIZATIONS.
ON THE SEMANTICS OF INTERACTIVE VISUALIZATIONS.
AN OPERATOR INTERACTION FRAMEWORK FOR VISUALIZATION SYSTEMS
A TAXONOMY OF VISUALIZATION TECHNIQUES USING THE DATA STATE REFERENCE MODEL
INFORMATION VISUALIZATION AND VISUAL DATA MINING
BEYOND MOUSE AND KEYBOARD: EXPANDING DESIGN CONSIDERATIONS FOR INFORMATION VISUALIZATION INTERACTIONS
INTERACTION SPACES IN DATA AND INFORMATION VISUALIZATION
AN INTERACTION VIEW ON INFORMATION VISUALIZATION
LOW-LEVEL COMPONENTS OF ANALYTIC ACTIVITY IN INFORMATION VISUALIZATION
TOWARD A DEEPER UNDERSTANDING OF THE ROLE OF INTERACTION IN INFORMATION VISUALIZATION
A FRAMEWORK OF INTERACTION COSTS IN INFORMATION VISUALIZATION
SPATIAL REASONING WITH EXTERNAL VISUALIZATIONS: WHAT MATTERS IS WHAT YOU SEE, NOT WHETHER YOU INTERACT
THE SCIENCE OF INTERACTION
CHARACTERIZING USERS’ VISUAL ANALYTIC ACTIVITY FOR INSIGHT PROVENANCE
MENTAL MODELS, VISUAL REASONING AND INTERACTION IN INFORMATION VISUALIZATION: A TOP-DOWN PERSPECTIVE
FLUID INTERACTION FOR INFORMATION VISUALIZATION
SEMANTIC INTERACTION FOR VISUAL TEXT ANALYTICS
INTERACTIVE DYNAMICS FOR VISUAL ANALYSIS : A TAXONOMY OF TOOLS THAT SUPPORT THE FLUENT AND FLEXIBLE USE OF VISUALIZATIONS
AN EMPIRICALLY-DERIVED TAXONOMY OF INTERACTION PRIMITIVES FOR INTERACTIVE CARTOGRAPHY AND GEOVISUALIZATION
A MULTI-LEVEL TYPOLOGY OF ABSTRACT VISUALIZATION TASKS
INTERACTION DESIGN FOR COMPLEX COGNITIVE ACTIVITIES WITH VISUAL REPRESENTATIONS: A PATTERN-BASED APPROACH
A DESIGN SPACE OF VISUALIZATION TASKS
BRUSHING SCATTERPLOTS
STARTING SIMPLE: ADDING VALUE TO STATIC VISUALISATION THROUGH SIMPLE INTERACTION
READINGS IN INFORMATION VISUALIZATION : USING VISION TO THINK
ILLUMINATING THE PATH: THE RESEARCH AND DEVELOPMENT AGENDA FOR VISUAL ANALYTICS
A TAXONOMY OF TEMPORAL DATA VISUALIZATION TECHNIQUES ???? (NOT SURE HOW THIS PAPER APPEARED)
A PROPOSAL FROM THE POINT OF VIEW OF INFORMATION VISUALIZATION AND HUMAN COMPUTER INTERACTION FOR THE VISUALIZATION OF DISTRIBUTED SYSTEM LOAD
INFORMATION VISUALIZATION : DESIGN FOR INTERACTION
VISUAL PERCEPTION AND MIXED-INITIATIVE INTERACTION FOR ASSISTED VISUALIZATION DESIGN
STACK ZOOMING FOR MULTI-FOCUS INTERACTION IN TIME-SERIES DATA VISUALIZATION
ADVANCED INTERACTION FOR INFORMATION VISUALIZATION.
INTEGRATING VISUALIZATION AND INTERACTION RESEARCH TO IMPROVE SCIENTIFIC WORKFLOWS
AN EXPLORATORY STUDY OF INTERACTIVITY IN VISUALIZATION TOOLS:'FLOW'OF INTERACTION
FI3D: DIRECT-TOUCH INTERACTION FOR THE EXPLORATION OF 3D SCIENTIFIC VISUALIZATION SPACES
EXPLORING INFORMATION VISUALIZATION: DESCRIBING DIFFERENT INTERACTION PATTERNS
ANALYSTS AREN'T MACHINES: INFERRING FRUSTRATION THROUGH VISUALIZATION INTERACTION
CARTOGRAPHIC INTERACTION PRIMITIVES: FRAMEWORK AND SYNTHESIS
AN INTERACTION MODEL FOR VISUALIZATIONS BEYOND THE DESKTOP
REIMAGINING THE SCIENTIFIC VISUALIZATION INTERACTION PARADIGM
MULTILEVEL INTERACTION MODEL FOR HIERARCHICAL TASKS IN INFORMATION VISUALIZATION
INTERACTION IN THE VISUALIZATION OF MULTIVARIATE NETWORKS
DECLARATIVE INTERACTION DESIGN FOR DATA VISUALIZATION
MOVEXP: A VERSATILE VISUALIZATION TOOL FOR HUMAN-COMPUTER INTERACTION STUDIES WITH 3D PERFORMANCE AND BIOMECHANICAL DATA
AN INTERACTION FRAMEWORK FOR LEVEL-OF-ABSTRACTION VISUALIZATION OF 3D GEOVIRTUAL ENVIRONMENTS
ADAPTIVE VISUALIZATION INTERFACE THAT MANAGES USER'S COGNITIVE LOAD BASED ON INTERACTION CHARACTERISTICS
INTERACTION FOR VISUALIZATION SYNTHESIS LECTURES ON VISUALIZATION
TOWARDS THE UNDERSTANDING OF INTERACTION IN INFORMATION VISUALIZATION
QUERY2QUESTION: TRANSLATING VISUALIZATION INTERACTION INTO NATURAL LANGUAGE
NATURAL INTERACTION WITH VISUALIZATION SYSTEMS
A VISUALIZATION-ANALYTICS-INTERACTION WORKFLOW FRAMEWORK FOR EXPLORATORY AND EXPLANATORY SEARCH ON GEO-LOCATED SEARCH DATA USING THE MEME MEDIA DIGITAL DASHBOARD
EVALUATION OF TWO INTERACTION TECHNIQUES FOR VISUALIZATION OF DYNAMIC GRAPHS
A CASE STUDY USING VISUALIZATION INTERACTION LOGS AND INSIGHT METRICS TO UNDERSTAND HOW ANALYSTS ARRIVE AT INSIGHTS
FLEXIBLE ORGANIZATION, EXPLORATION, AND ANALYSIS OF VISUALIZATION APPLICATION INTERACTION EVENTS USING VISUAL ANALYTICS
SPATIALVIS: VISUALIZATION OF SPATIAL GESTURE INTERACTION LOGS
IVORPHEUS 2.0-A PROPOSAL FOR INTERACTION BY VOICE COMMAND-CONTROL IN THREE DIMENSIONAL ENVIRONMENTS OF INFORMATION VISUALIZATION
VISUALIZATION BY DEMONSTRATION: AN INTERACTION PARADIGM FOR VISUAL DATA EXPLORATION
VISUALIZATION AND INTERACTION WITH MULTIPLE DEVICES. A CASE STUDY ON REACHABILITY OF REMOTE AREAS FOR EMERGENCY MANAGEMENT
AUTHORS
TITLE
YEAR
LOW HIGH
0
1
2
SEED PAPERSLINKED PAPERS
TAGS
METRICS
TAGS
METRICS
CONCEPT
PIPELINE
RELEVANCE P
FORWARD P
EXPERT INPUT
PC (citations)
IMPACT P
BACKWARD P
0 12
CRITICAL
BENEFITS
DEFINITION
1991
1996
1996
1998
2000
2002
2003
2004
2004
2005
2007
2008
2008
2009
2009
2010
2011
2012
2012
2013
2013
2013
2013
1987
1998
1999
2005
2005
2005
2007
2008
2010
2010
2010
2010
2010
2010
2011
2012
2013
2013
2013
2014
2014
2014
2014
2014
2015
2015
2015
2015
2015
2016
2016
2016
2016
2016
2017
2017
CARD ET AL.
SHNEIDERMAN
CHUAH AT AL.
CHI & RIEDL
CHI
KEIM
LEE ET AL.
WARD & YANG
KOSARA ET AL.
AMAR ET AL.
YI ET AL.
LAM
KEEHNER ET AL.
PIKE ET AL.
GOTZ & ZHOU
LIU & STASKO
ELMQVIST ET AL.
ENDERT ET AL.
HEER & SHNEIDERMAN
ROTH
BREHMER & MUNZNER
SEDIG & PARSONS
SCHULZ ET AL.
BECKER & CLEVELAND
DIX AND ELLIS
CARD ET AL.
COOK & THOMAS
DAASSI ET AL.
LARREA ET AL.
SPENCE
HEALEY ET AL.
JAVED & ELMQVIST
FEKETE
KEEFE
LIANG ET AL.
YU ET AL.
POHL ET AL.
HARRISON ET AL.
ROTH
JANSEN & DRAGICEVIC
KEEFE & ISENBERG
REN ET AL.
WYBROW ET AL.
SATYANARAYAN ET AL.
PALMAS ET AL.
SEMMO & DÖLLNER
YELIZAROV & GAMAY
TOMINSKI
FIGUEIRAS
NAFARI & WEAVER
AMANT
SJÖBERGH ET AL.
FEDERICO & MIKSCH
GUO ET AL.
HAN ET AL.
PALUKA & COLLINS
FURTADO ET AL.
SAKET ET AL.
TOST & HEIDMANN
LIST
Fig. 1. The 23 seed and 36 linked papers in our review, ordered by year.
It shows expert input, the algorithm metrics, and our tagging scores.
being who can be characterized by skills [36], abilities, expertise and
motivation [99], and who initiates the interaction [104]. Some papers
differentiate between end-users and designers [22]. The data is an
intangible information source that is the user’s main object of interest.
The mean with which the user interacts with the data is a mediating
[98, 99] entity that we call the visualization system. The visualization
system (either as a whole or some of its components) is referred to as
technology [74, 99, 104], computational tools [92] and computer [8]
with hardware and software [8], when emphasizing its technological
properties; representation [76,104,126], visualization [16,19,22,45,67,
68,72,79,99,120], map [98], graphical marks [2,22], display [8,74,106],
visual metaphor [37] and non-static image [70], when emphasizing its
visual properties; and interface [19, 22, 36, 76, 92, 98,99, 104], system
[19, 36, 45, 62, 74, 123, 126], analysis tool [92], workspace [19] and
interaction space [104], when referring to a more abstract mediating
entity. Some papers add that the mediating entity should provide a set
of controls [92] for the user to access the data.
Most papers identify time as a fundamental entity of interaction [16,
62, 76, 126]. Interaction, unlike representation [126], is seen as a real-
time [8, 98] action with a start and an end [74] that should result in an
immediate response from the visualization system [16, 19, 36, 106, 123].
INTERNAL ENTITIES: Interaction is characterized as a goal-oriented
activity [53, 62, 92] involving a data-oriented intent. Intent [36, 45,
74, 76, 79, 84, 92] is also phrased as goal [45, 62, 76, 92, 97
–
99, 123],
task [45, 67, 106] or problem [53, 92]. Intent can describe a high-level
data exploration [19, 45, 62, 68, 84, 92,126], the generation of insights
[45, 62, 92, 98, 99, 126], and the need to acquire multiple perspectives
on the data [62,92,104, 126]. Intent can go beyond exploration, such as
the intent to collect and correct data [62], or social intents to coordinate
in collaborative setups and to present data to an audience [62]. Intent is
both identified at a low, operational level, e.g., to alter the representation
[19, 126], and at a higher level, e.g., for information foraging [79],
sensemaking [99, 104], and knowledge creation [92].
EXTERNAL ACTIONS: Interaction is described as a dialogue between
the user and the visualization system [36,76,92,98, 104, 123,126] made
of action-reaction pairs [76, 104]. The user performs an action (or
input [22,102]) on the visualization system [45,67,72,76, 79, 104]. The
visualization system returns a reaction [74,76,104] (or response [19,67,
74,76,98,106,120,126], change [8,16,22,98,120,126], output [22,102])
that is perceived by the user [72, 104, 120]. This reaction has been
called permutation of graphical marks [2], change of transformation
parameters [123], and alteration of the pipeline [62].
INTERNAL ACTIONS: Along with its physical acts, interaction with a
visualization system involves a cognitive act of the user [37, 92, 104] or,
similarly, a reasoning/analytic process [67, 76, 79,97] on the data.
3.1.2 What Interaction for Visualization Can Involve
Interaction for visualization can involve additional external and internal
entities, and additional external and internal actions.
EXTERNAL ENTITIES: Interaction can involve external physical objects
[62, 74, 106] such as mouse [79], pen [74] keyboard [79] and physical
constraints [99]. It can also involve a variety of modalities such as
body movements [19,74], speech [74], head [79] and eye movement
[79]. Many modalities (e.g., gaze, head) tend to be overlooked in
visualization. Restricting modalities leads to a “disjointed picture of
human performance” [79] and lost opportunities to capture user intent
[79]. Interaction also involves the environment under which interaction
occurs, for example whether it is a casual or working environment [99],
and whether there are multiple users involved [62, 74, 92, 99].
INTERNAL ENTITIES: Users’ prior knowledge [92], internal mental
representation [104], skills [36] and abilities [99] can be involved when
interacting with a visualization system. While interaction can start
with a concrete user query [19, 45, 84, 106], it can also lead to the
internalization of new goals [104]. Moreover, interactions might occur
with an absence of intent, such as with proxemics interactions [74].
EXTERNAL ACTIONS: Interaction can involve many user actions. Such
action can change the data [16, 22, 45,76, 84,92, 102,104], e.g., with
filtering and aggregation [102]. It can change the representation and
presentation of information [8, 16, 19,37, 45,68, 79,84, 104], e.g., with
sorting [45] or when switching from a map to a timeline [45]. It
can create metadata, for example by temporarily marking data to track,
annotate or bookmark [37,45,84]. It can create new data representations
[74, 79] and new data, to express and manipulate new knowledge [37,
45, 104], e.g., with note taking and when manipulating a knowledge
management component [45]. A user can perform an action that does
not occur on the data or their representation, e.g. adjusting a movable
baseline to compare the heights of a histogram [126], performing a
metaction on their own action history, (undo/redo) [45], and steering a
statistical data model [37]. Last, along with the dialogue between the
user and the visualization system, there is also the dialogue of users
with the external environment [92] and with their peers [62, 74, 92, 99].
INTERNAL ACTIONS: While interaction with a data source is enabled
within the context of a tangible visualization system, much of it can
occur internally in user’s mind [74, 92], for example information pro-
cessing [104], memory encoding [104], and simulative reasoning [79].
On a deeper level, the interaction takes the form of a dialogue between
the “internal representations and processes of the user and the external
representations and processes” of the visualization system [104].
3.1.3 Properties of Interaction
Interaction for visualization is characterized as a goal-oriented [62,
76, 92, 97
–
99, 123] activity that contains semantics [22, 45], that is
sequential [22, 45, 98, 104], incremental [16, 36,37, 106], and iterative
[19, 98, 104], and preserves the following properties:
GRANULARITY: Interaction is characterized at multiple levels of gran-
ularity [2, 45, 62, 76, 102, 104]. Such levels include micro-level (e.g.,
mouse clicks), macro-level (e.g. hypothesis generation) and levels in
between (e.g., filter, sort). To distinguish these levels, terms such as
{
subtasks, tasks, activities
}
[2],
{
events, actions, subtasks, tasks
}
[45]
and
{
events, streams, signals, predicates
}
[102] have been used. Yet,
it is often unclear whether these levels refer to interaction per se or to
user intent. The lack of distinction between levels of interaction results
in no established conceptualization and vocabulary [104].
CONTINUOUS VS. DISCRETE: The temporal dimension, which is a
fundamental entity of interaction (see Sect. 3.1.1), is either continuous
or discrete [76, 84]. Continuous interactions involve a sequence of
intermediate visualization states between an initial state and a goal state
(e.g., mouse drags to perform a lasso selection) [84]. Such interactions
happen over a span of time [76]. With discrete interactions, action and
reaction occur in a distinct manner [76] (e.g., clicking on a checkbox to
toggle filtering [84]). Action and reaction can have different continuity
(e.g., a continuous action can have a discrete reaction [76]).
DIRECT VS. INDIRECT: Directness is associated with continuous repre-
sentation of the objects of interest, rapid, incremental, and reversible
user actions; allowing for usage with minimal knowledge [16, 36]. Di-
rectness was first associated with techniques such as dynamic queries
[16] that provide widgets to explore the data interactively – as opposed
to command-line interfaces. But the meaning of directness has changed
and dynamic queries are now said to be indirect because the user in-
teracts with intermediate, likely spatially distant widgets – as opposed
to interacting directly with the representation of the data itself. Rather
than (quite arbitrarily) classifying interactions as either direct or indi-
rect, it is useful to consider that interactions have degrees of directness
on an indirect–direct continuum [62].
3.1.4 Terms Related to Interaction
Interaction is distinct from interaction technique, interactivity, and sci-
ence of interaction. Interaction technique is less broad [104] and more
tangible [126] than interaction, and refers to the user means with which
interaction can occur [92] (in terms of hardware and software [62]). Un-
like interaction, an interaction technique does not necessarily embed the
notion of intent [92]. Whereas interaction refers to the action-reaction
dialogue between a user and visualization system, interactivity refers to
the feel, properties, and quality of this interaction [76]. Yet, the terms
are sometimes used interchangeably [21, 62]. Science of interaction is
broader than interaction [92]. It is the study of methods by which hu-
mans create knowledge through interaction, and it involves developing
and testing theories and practices to better support interaction [92]. All
these terms internalize the notion of interaction. Thus, making them
actionable requires a precise view of what interaction is.
3.2 Reported Benefits & Critiques on Interaction
39 papers reveal benefits of interaction. At first, interaction was seen as
a necessity to handle increasing amounts of data [32, 38, 64, 106, 126].
But interaction has moved beyond its necessity. It is now seen as a
mean to amplify cognition in active, human-driven data exploration
[37, 47, 54, 68, 70,79, 95, 104,125, 126] in which the user is in control
of the information space [36,100, 106]. It is via interactive manipula-
tion that “knowledge is constructed, tested, refined and shared” [92].
Further, interaction leverages humans’ natural abilities through new
visualization shapes, modalities, and input technologies [36, 62, 74, 77],
helping to make visualizations accessible to broader audiences [74].
Despite these benefits, the 36 critical papers highlight frustrations
around the topic of interaction. The most frequent one is that in compar-
ison to representation, interaction is rarely the focus of research efforts
in visualization [2, 21, 36, 38, 72, 74, 99,126]. When interaction is the
focus of research, the angle is often more on engineering or implemen-
tation than on designing for interaction [36, 99] and facilitating user
analytic activities [2]. A key concern is the limited consideration for
human (e.g., gaze, speech, haptics, sound, full-body) and technology
modalities (e.g., pen, sketch, multi-touch surfaces), compared to the
classic desktop-mouse-keyboard setups [36, 54, 65, 66, 74]. Beyond
modalities, visualization systems are often not flexible when users want
to express complex data queries [16, 54, 66, 120], integrate annota-
tions [39, 54], input new data [21], bookmark and extract insights [39],
iterate over their activity history [39, 84], organize freely elements of
the layout [54], choose their own statistical [37,54] and visualization
models [54, 79, 100, 102], and collaborate in real-time [39, 54].
This lack of flexibility stems from several factors, including tech-
nical challenges [90, 102], costs [72] and the lack of theoretical foun-
dations that bridge external representations with internal cognitive
processes [47, 79, 104]. Two other factors are seemingly conflicting.
On one hand, there is the failure of visualization systems to infer user
goals. This is attributed to a strong focus from the community on
data, tasks [14, 72] and domains [104] rather than on human goals;
and to a difficulty to infer user goals via activity logs [51, 76, 84]. On
the other hand, there is an overfocus on operationalizing user goals.
As a result the visualization community ignores other objectives that
broaden the interaction palette, such as engagement, playfulness and
gamification [36, 39, 74], which reflect softer, or even lack of, user goal.
3.3 How Visualization Conceptualizes Interaction
We identified the need for more flexible interactions with respect to
the outside world (e.g., environment, social aspect, modalities) and to
the tool itself (e.g., user input, fluid interface permutation). We also
identified the need to resolve the ambiguity between intent and lack of
flexibility. Next, we review papers with a concept or a list tag to learn
(i) how they conceptualize human intent, and (ii) how a visualization
system can account for such intent to offer flexibility to the user.
3.3.1 System, Task or Human?
The concept and list papers reveal three approaches to conceptualize
interaction for visualization: system-, task-, and human-centric.
System-centric approaches describe interaction in terms of opera-
tors [20
–
22, 28, 120]. These operators have been adapted to specific
domains such as temporal visualizations [28] and visualizations of 2D
data [99], and have been organized in a network [20] and in a hierar-
chy [22]. Focusing on system operations, system-centric approaches
do not clarify neither the role of the user nor how the user interacts.
Task-centric approaches describe interaction in terms of low level
tasks (also primitives or components) [2, 99, 103]. Such low-level
tasks have been put in relation with data types [106], means, target,
cardinality, (temporal) order, and user roles (developers, authors, end
users) [103]. It has been argued that high-level visualization tasks build
on low-level interactions [66], but the relations and boundaries between
low, intermediate, and high-level interactions are ill-specified [14, 112].
Human-centric approaches describe interaction in terms of user in-
tent [32,39,47,99,126] rather than low-level mechanisms for supporting
these intents. High-level cognitive activities introduce interactions such
as notes (external source of data and external knowledge) and history
that relate to insight actions [45], meta actions [45], and provenance
actions [54], as opposed to exploration actions. Most human-centric ap-
proaches assume a high-level user intent in a data analysis context, such
as making profitable investments with stock market data [45] or address-
ing the low productivity of a virtual factory network [97]). However, a
few human-centric approaches expand the scope of user intent [79] by
considering factors of interaction beyond data, system and tasks. Such
factors include the environment (e.g., academic, museum), the context,
the technology, the domain, the audience (e.g., web, analyst, casual),
the task (e.g., exploration, immersion), and the properties of interaction
(e.g., direct, embodied, aesthetic, rewarding) [36, 112]. Considering
these factors builds higher-level views of interaction, that relate for
example to mental models [79] and epistemic actions that take place
during sense making, problem solving and decision making [104].
The broader the approach in terms of user intent, the less clear it
is how a visualization system can operationalize the flexibility that is
needed to accommodate this wider spectrum of user intents.
3.3.2 Interaction & Visualization Pipeline
One of the most prominent concepts that describes a visualization sys-
tem as a whole, from user’s high level goal to low level data operations,
is the visualization reference model, or visualization pipeline [16].
However, only 12/59 papers discuss the role of interaction in it. Most
of those papers [16,20,21,25,28,62,98] stress that interaction can affect
all levels of the pipeline i.e., transformations of raw data, processed
data, visual mappings and view. A few papers organize transforma-
tions [100] or interaction techniques [123] according to the level of the