IHMC Technical Report
IHMC Technical Report, October 2012
1
Coactive Emergence as a Sensemaking Strategy for
Cyber Operations
Jeffrey M. Bradshaw
1,
*, Marco Carvalho,
2
Larry Bunch
1
, Tom Eskridge
1
, Paul J. Feltovich
1
,
Chris Forsythe
3
, Robert R. Hoffman
1
, Matt Johnson
1
, Dan Kidwell
4
and David D. Woods
5
1
Florida Institute for Human and Machine Cognition (IHMC), 40 South Alcaniz Street, Pensacola, FL 32502
2
Department of Computer Science, Florida Institute of Technology (FIT), Melbourne, FL
3
Sandia National Laboratories, Albuquerque, NM
4
U.S. Department of Defense, U.S. Government
5
The Ohio State University, Columbus, OH
Abstract
In this article we describe how we apply the concept of coactive emergence as a phenomenon of complexity that has
implications for the design of sensemaking support tools involving a combination of human analysts and software agents.
We apply this concept in the design of work methods for distributed sensemaking in cyber operations. Sensemaking is a
motivated, continuous effort to understand, anticipate, and act upon complex situations. We discuss selected results of a
macrocognitive work analysis that informed our focus for design and development of support tools. In that analysis, we
identified seven target topics that would be the focus of our research: engaging automation as a full partner, reducing the
volume of uncorrelated events, continuous knowledge discovery, more effective visualizations, collaboration and sharing,
minimizing tedious work, and architecting scalability and resilience. In addressing the first target topic, we show how
coactive emergence inspires an agent-supported threat understanding process that is consistent with Klein’s Data/Frame
theory of sensemaking. In subsequent sections, we describe our efforts to address the remaining six target topics as part of
design and development of a cyber operations framework called Sol. Specifically, we describe the use of agents, policies,
and visualization to enable coactive emergence for taskwork and teamwork. We also show how policy-governed agents
working collaboratively with people can help in additional ways. We introduce the primary implementation frameworks
that provide the core capabilities of our Sol cyber framework: the Luna Software Agent Framework, and the KAoS Policy
Services Framework. We describe areas for future development of Sol, including the incorporation of the VIA Cross-Layer
Communications Substrate. Finally, we describe recent results and current plans for empirical studies addressing some of
the issues raised in this article.
Keywords: cyber defense, cyber operations, cyber security, teamwork, software agents, policy management, organic resilience, coactive
emergence, sensemaking
*
Corresponding author. Email:jbradshaw@ihmc.us
1. Introduction
In broad terms, the work of the cybersecurity
professional, on behalf of their organization, is to
formulate answers and undertake actions in response to
questions such as the following:
• What is the nature and purpose of current attacks
and what is their origin?
• What are the attackers doing now and what might
they do next?
• How do the attacks affect my mission now and how
might they affect it in the future?
• What options do I have to defend against these
attacks?
• How effective will a given option be against these
attacks and what effect will exercising it have on my
mission and how is it likely to affect the future actions
of allies and adversaries?
• How do I prevent or mitigate the impact of such
attacks in the future?
Analysts working in large-scale Network Operations
Centers (NOCs) are a vital part of cyber defense as they
monitor, detect, understand, and respond to attacks or
other conditions (e.g., power failures) that might impact
mission performance. Typically working in close
proximity within large rooms filled with individual
workstations and a video wall at the front intended to
keep everyone aware of important developments that may
affect their work, they are organized into hierarchical
groups with different duties or spans of responsibility.
Some analysts are more focused on ongoing monitoring
of events at the moment-to-moment level, while others are
responsible for strategic direction or in-depth analysis of
serious incidents.
Despite the significant attention being given to the
critical challenges of cyber operations within large-scale
NOCs, the ability to keep up with the increasing volume
J. M. Bradshaw, M. Carvalho, L. Bunch, T. Eskridge, P. Feltovich, C. Forsythe, R. R. Hoffman, M. Johnson, D. Kidwell, D. D. Woods
2
and sophistication of network attacks is seriously lagging.
Cyber defense, by its very nature, is asymmetrically
disadvantaged in its efforts to fend off attackers and the
perception by most of the experienced analysts we have
encountered is that the imbalance is worsening. While
attackers can strike at their leisure and can profit from the
careless exposure of virtually any vulnerability, defenders
must be continually vigilant and responsive—both
proactively and reactively—to potential threats relating to
any aspect of their systems.
Merely throwing more computing horsepower at
fundamentally limited visualization and analytic
approaches will not advance our aims. Extensive
experience in domains with similar challenges has shown
that the kinds of complex automation often seen in NOCs
today do not adequately leverage human creativity,
ingenuity, and flexibility—besides actually hindering
analyst effectiveness in some ways. Though ongoing
efforts to increase computing resources and improve
technology is essential, the point of providing these
enhanced proficiencies is not merely to make
computational tools more capable in and of themselves,
but also to make analysts more capable through the use of
such technologies [75]. To better empower these
professionals, we need to seriously rethink the way cyber
operations tools and approaches have been conceived,
developed, and deployed.
In this article, we focus on selected problems for
distributed sensemaking and response in Cyber Defense
Analysis and other roles in cyber operations. In particular,
we describe our experiences in applying knowledge about
the cognitive sciences to help analysts working in large-
scale NOCs. Though it will be impossible in this article to
discuss more than a sampling of relevant research, we will
survey some concepts and findings running the gamut
from basic cognitive science (e.g., perception, attention,
inference, individual differences) to socio-cognitive issues
(e.g., theories of social interaction, human-automation
teamwork).!
As rationale for the principles used in our work design,
we present the results of a macrocognitive work analysis
(Section 2). In that analysis, we identified seven target
topics that would be the focus of our research: engaging
automation as a partner, reducing the volume of
uncorrelated events, continous knowledge discovery,
more effective visualizations, collaboration and sharing,
minimizing tedious work, architecting for scalability and
resilience. In addressing the first target topic, we describe
the Klein, et al. Data/Frame theory of sensemaking and
introduce the concept of coactive emergence. In
subsequent sections, we describe our efforts to address the
remaining six target topics as part of the design and
development of a cyber operations framework called Sol
(Sections 4-9). Specifically, we describe the use of
software agents, policies, and visualization to enact a
sensemaking strategy for taskwork and teamwork inspired
by the phenomenon of coactive emergence. We also show
how policy-governed agents, working in tandem with
people, can help in additional ways. We introduce the
primary implementation frameworks that provide the core
capabilities of our Sol cyber framework: the Luna
Software Agent Framework, the VIA Cross-Layer
Communications Substrate, and the KAoS Policy Services
Framework. Finally, we describe results of empirical
studies addressing some of the issues raised in this article
(Section 10), as well as anticipated trajectories for future
development of the Sol framework (Section 11).
2. Macrocognitive Work Analysis
Macrocognitive work is how cognition adapts to
complexity [5]. Distinguished from the phenomena of
cognition that are studied in the traditional psychology
laboratory, macrocognition includes such functions as
sensemaking, adapting, and collaborating. The study of
macrocognitive work involves methods of cognitive task
analysis, although we recognize that the term “task,” as it
is traditionally used, is less apt than the term “work.”
2.1. Approach
For the project that we report here, we engaged in a
literature survey, obtrusive workplace observations,
participation and discussions as part of training exercises,
and semi-structured interviews, case study reviews, and
discussions with cyber defense analysts in government
and private industry. Concept maps, text notes, and
drawings were used to record our sessions, however no
formal methods of knowledge modeling were used and no
formal analysis of the results was undertaken.
Our approach was oriented around four major kinds of
inquiries:
1. Finding out what aspects of the work-shaping
technologies were most important yet caused the
most difficulty. Of prime value to gaining an
understanding of the analyst’s work and its
requirements was to understand what activities are
the most important for conducting work effectively
and why. We tried to learn which of these important
activities were the most difficult to manage or
overcome. Subsequently, we explored some of the
perceived reasons for this difficulty. This kind of
exercise starts to give us focus in our inquiries and
research directions, in order to assure that we are
working on problems of high value [1]. We call these
areas of interest “target topics.”
2. Inquisitive observation of practice and discussion of
case studies to understand the “actual work.” We
supplemented our observation of experts through
readings and discussions of case studies and work
practices. In addition to studying guidelines for
standard operations, we have been interested in
deviations from these expected practices, the
presence of “invisible” (vs. overt) work, and
contextual adaptations in the face of field expediency
[2]. We reviewed case studies with experts under a
Coactive Emergence as a Sensemaking Strategy for Cyber Operations
3
modified “think aloud” procedure. That is, we asked
analysts to tell us generally what they were doing at
different stages of the activities being reviewed, and
we were able to ask questions as their activities
interacted with particular points that we were trying
to understand. We paid particular attention to any
encounters with the “target topics.”
Of particular interest are cases that may be seen as
challenging analysts for reasons such as the
following: 1. they taxed the limits of their expertise
(e.g., the solving of an analytic “puzzle”); 2. they
required various workarounds (e.g., technology gaps;
organizational or procedural inconveniences that
necessitated “extra” steps in the work); or 3. they
raised personal, organizational, or policy dilemmas
(e.g., situations where simply following the accepted
procedure would have produced an unacceptable
result, or where invisible or explicit organizational
and policy structures created barriers to effective
performance). Such inquiries identify leverage points
for technological interventions, and reveal ineffective
problem-solving strategies that affect individual work
performance and collaboration (see, e.g., [3]).
3. Finding out the analysts’ “desirements” [76], that is,
functionalities and features they would like to have
that would make it easier for them to achieve their
work goals. We conducted additional structured
discussions on specific questions with analysts to get
feedback on design ideas that the team had generated.
These discussions continued throughout the project,
feeding a spiral development process on the major
technological capabilities developed.
4. Creation and refinement of a scenario as part of the
quest for generalizability. Based on information
gleaned from the activities described above, we
created a detailed scenario of a 24/7 network
operations context. The scenario provided a narrative
that would illustrate, and qualitatively represent, the
policy-driven, agent-based monitoring and control
capabilities being developed. The scenario was
reviewed, discussed, and refined with project
sponsors, with professional colleagues, and with
practicing analysts. Discussions of the scenario
helped reveal hidden requirements and concerns that
were not always revealed directly by the work
analysis itself.
Cognitive engineering approaches of this sort entail a
level of complexity and nuance that is not encountered in
more traditional classroom or laboratory studies.
However, because of the broader range of issues
considered in our “field research” approach, we believe
that it is more likely than laboratory experimentation to
reveal underlying factors that will enable recommended
improvements in organizational, policy, and work systems
design, and would enable technology support to have a
more powerful, predictable, and lasting impact.
2.2. Target Topics and “Desirements”
Among the target topics (challenges to the
macrocognitive work) that emerged from our observations
and discussions were the following. Most of these are
specific instances of problems that were actually created
when tool developers took a designer-centered rather than
a human-centered approach to design:
1. Engaging automation as a partner in the rapidly-
evolving process of sensemaking and response.
Analysts are accustomed to using a piecemeal set of
software tools in the accomplishment of their work,
pulling out a software “wrench” when a wrench was
called for, and a software “hammer” when a hammer
was called for. Each tool had been designed to
perform one or more specific, generic tasks, but no
tool really “understands” the overall work in which
the analyst might be engaged. It was people who
provided the know-how needed to use the tools, the
sometimes-arcane routines needed to transfer data
among them, and, most importantly, the
understanding of the overall context and objectives
that motivated and shaped the effort. When the tools
were not merely passive, they were seen as
adversarial—targets of pointed cursing because of
their limitations (a phenomenon called “automation
abuse” [77]).
The dream of analysts was not a toolset, but a
software teammate that would understand something
about what they were trying to do and could actively
assist them in overall sensemaking and response
processes—both teaching them and being taught in
an iterative process of mutual interdependence. Could
today’s stove-piped tools be integrated into a context-
sensitive, task-aware, and assistive capability? A
related problem is that both the nature of attacks and
the details of work practice inevitably change much
more rapidly than the traditional software
development and release cycles currently support.
Would it be possible to build technologies that could
evolve as quickly as threats and responses do? Could
a system be made to straightforwardly assimilate
future analytic and response innovations that cannot
presently be anticipated? Could the tools for creating
that new work system be made simple and yet
adaptive enough such that analysts could use them in
do-it-yourself fashion?
2. Reducing the great volume of uncorrelated low-level
events. Analysts tasked with monitoring and
performing triage on network events can be
overwhelmed by the massive volume of uncorrelated.
low-level, and simplistic alerts and alarms with which
they were continuously confronted. Analysts asked
for better tools for the detection of complex
anomalies, especially those that are context-specific
or involve correlations across multiple data sources.
They wanted help in understanding history and
trends, so they could better understand what was
J. M. Bradshaw, M. Carvalho, L. Bunch, T. Eskridge, P. Feltovich, C. Forsythe, R. R. Hoffman, M. Johnson, D. Kidwell, D. D. Woods
4
normal and recognize when significant long-term or
short-term deviations in expected findings are taking
place.
3. Enabling continuous knowledge discovery and
enrichment. Analysts continually divide their time
among a multitude of tasks. Their work in pursuing a
given objective may be interrupted for hours or days
while they deal with a sudden emergency. Tools that
could continue to monitor relevant data sources in
their absence, enrich results with pertinent
information (e.g., geographic localization, entity
identification and elaboration, database correlations),
and organize those results on their own for later
review by the human analyst were seen as having
great potential.
4. Overcoming the inadequacies of visualization tools.
Visualization tools were seen as inadequate in several
respects. One problem is scalability. For example,
parallel coordinate displays are not intelligible for
any more than a few dozen network traffic records.
Another problem was the form and content of what
was presented. For instance, dashboard-style displays
do not present information of different types in an
integrated and meaningful fashion that directly
answers analyst questions of central interest. Displays
are typically technology-centered—focusing on what
can easily be shown—rather than human-centered—
focusing on what needs to be known. Display designs
are fatiguing rather than appropriately stimulating to
the eye and the imagination because they do not
reflect sensitivity to issues of human perception and
cognition.
Another issue is a lack of interactivity—effective
sensemaking requires not just “seeing” the data but
also being able to probe and interact with it—and, in
addition, requires the capability for the analyst to take
action when necessary without having to move to a
different display or software application. Displays are
typically retrospective, showing something that had
happened, rather than helping analysts anticipate
what might happen next through the extrapolation of
current trends, and assisting them in taking proactive
measures when appropriate.
5. Encouraging collaboration and sharing across
individuals and distributed groups. They face a
plethora of information sharing challenges that
sometimes lead to critical failures in achieving the
common ground needed for understanding and
effective action. First, analysts were sometimes
unaware that information they possessed could be
useful to someone else, or vice versa. Second,
analysts are limited to specific means of
communication (e.g., phone calls, chats) that can
make it difficult and time-consuming to convey the
richness of their observations. Third, the simplistic
nature of today’s digital policy management systems
results in ambiguities about what could be shared
with whom, and sometimes leads to out-of-band
workarounds to circumvent inflexible systems when
all else failed. Fourth, and most fundamentally,
shared visualizations, such as those that might appear
on large displays at the front of a room housing a
NOC, have generally suffered from a lack of careful
study of what kinds of information might actually be
useful in such contexts.
6. Minimizing the burdens of tedious everyday work.
Analysts complained about the amount of tedious and
time-consuming work, including writing of a variety
of report types. Awkward adaptations have
proliferated as means to manage their burdens and to
deal with the rigidity of tools and procedures. The
ability to assess the status and progress of ongoing
individual and group activities was sorely lacking.
The need for a means of capturing and sharing
knowledge with less-experienced analysts was
expressed. Related to this problem was the loss of
important “organizational memory” when analysts
left or retired or when a case was “finished.”
7. Architecting for scalability and resilience. Our
interviewees said that they imagined that future
analysts would need to be able to work securely and
effectively in increasingly heterogeneous computing
environments. Unfortunately, software systems are
not usually designed with this forward look in mind.
On the one hand, there is a need for a computing
architecture that can automatically scale to varying
computing and network resources. On the other hand,
new kinds of computing devices, large and small, will
continue to proliferate, and analyst will want to be
able to use and synchronize their information across
all of them. In addition, organizations will
increasingly expect their technological support
systems to be engineered for resilience, ensuring
mission continuity, even when under attack or
experiencing failures.
We are using the above target topics and “desirements”
to guide the design and development of a cyber operations
framework called Sol [4]. In the next sections we will
describe our efforts to address the first target topic:
engaging automation as a partner. In Sections 4-9, we will
do likewise for the other topics.
3. Engaging Automation as a Partner
With respect to our first target topic, the analysts we
interviewed were interested in engaging automation as a
partner in the process of sensemaking and response. In
order to lay the groundwork for a subsequent discussion
of the details of the design of Sol, we first give an
overview of what we mean by the term “sensemaking”
(Section 3.1). We outline the role of software agents as
partners in sensemaking (Section 3.2). We then introduce
the concept of coactive emergence (Section 3.3). In doing
this, we draw on the work of Johnson, who coined the
term “coactive design” as a way of highlighting
interdependence as the central organizing principle
Coactive Emergence as a Sensemaking Strategy for Cyber Operations
5
among people and agents working together [7][8][9][10].
We see coactive emergence both as a phenomenon of
complexity and also as a strategy for the design of
sensemaking work that combines the efforts of humans
and software agents
†
in understanding, anticipating, and
responding to unfolding events—both the foreseen and
the unforeseen.
3.1. Sensemaking
As defined by Klein, et al. ([5], p. 71), sensemaking “is
a motivated, continuous effort to understand connections
(which can be among people, places, and events) in order
to anticipate their trajectories and act effectively.”
Figure 1 illustrates what Klein and his colleagues call
the “data/frame theory of sensemaking” ([6], p. 89). At
the most basic level, the theory acknowledges that
understanding situations always occurs with respect to a
framing perspective. The frame constitutes a set of more
or less coherent hypotheses about the data to be
understood, and serves both to determine what counts as
data of interest and to shape the interpretation of the data.
Note the absence of input and output arrows in the
diagram. The sensemaking process can start, or
recommence at any point, even though it is often triggered
by surprise.
As data accumulate, the sensemaker may be confronted
with the question of whether to elaborate a current frame
by incorporating new details, or to seek a new frame that
better accounts for current findings. The process involved
†
In this article, the term “agent,” standing alone, will always refer to a
software agent. Likewise “analyst” will always refer to a human analyst.
in the ongoing evaluation of a given frame includes the
possibility of a closed-loop alternation between
backward-looking mental model formation—which seeks
to explain past events—and forward-looking mental
simulation—which anticipates future events.
The application of sensemaking concepts to the field of
intelligence analysis (e.g., [11]) has looked at the ways to
shape the sensemakers’ investigative procedures in order
to help them counteract lines of reasoning that might lead
to misconceptions. A basic foundation for analyst
sensemaking having been laid already in the research
literature, a next step is toward implementation of a
sensemaking support system that can harness the joint
power of humans and machines. In particular, an
understanding is needed of the potential impact of new
forms of visualization and automation on the sensemaking
process, and how such tools ought to be designed in light
of what we already know. The emphasis of our own work
on sensemaking is to put questions about the role and
benefits of computer interaction with people in center
stage.
In their discussion of the data/frame theory, Klein, et
al. conjecture that the role of machines in assisting people
with sensemaking may not be merely to confirm or
disconfirm the accuracy of a particular interpretation with
respect to a given frame, but also as an aid in the
reasoning process that leads to the possibility of
reframing: “The implication is that people might benefit
more from intelligent systems that guide the improvement
of frames than from systems that generate alternative
understandings and hypotheses and foist them on the
human” ([6], p. 89). This conjecture is consistent with the
view of Woods, et al., who have adopted a stance to
resilience engineering that takes as its basic assumption
that “human systems [are] able to examine, reflect,
Figure 1. The Data/Frame Theory of Sensemaking