Human behavior models for agents in simulators and games: part II: gamebot engineering with PMFserv

TL;DR: This paper explores whether the behavior modeling framework could embed behind a legacy first person shooter 3D game environment to recreate portions of the Black Hawk Down scenario and reveals that it was able to generate plausible and adaptive recreations of Somalian crowds, militia, women acting as shields, suicide bombers, and more.

Abstract: Many producers and consumers of legacy training simulator and game environments are beginning to envision a new era where psycho-socio-physiologic models could be interoperated to enhance their environments' simulation of human agents This paper explores whether we could embed our behavior modeling framework (described in the companion paper, Part I) behind a legacy first person shooter 3D game environment to recreate portions of the Black Hawk Down scenario Section I amplifies the interoperability needs and challenges confronting the field, presents the questions that are examined, and describes the test scenario Sections 2 and 3 review the software and knowledge engineering methodology, respectively, needed to create the system and populate it with bots Results (Section 4) and discussion (Section 5) reveal that we were able to generate plausible and adaptive recreations of Somalian crowds, militia, women acting as shields, suicide bombers, and more Also, there are specific lessons learned about ways to advance the field so that such interoperabilities will become more affordable and widespread

Summary

Introduction

• When the cooperation of multiple robots is required to complete a tightly-coupled task, the task is often referred to as a multirobot task (MT) [7].
• Both ASyMTRe and their work are inspired by information invariants theory [5].
• The weighted sensor quality measurements are taken as the information quality measurements.
• For extreme cases where certain sensor constraints are unsatisfiable, the authors define a constraint model that manages constraint repositories.

II. RELATED WORK

• Maintenance of sensor constraints is important in many applications.
• Moreover, uncertainty and dynamic factors can render the solution infeasible for practical applications.
• In [1], a greedy strategy for target tracking is introduced that uses local sensor data for geometric reasoning.
• The approach of [16] uses visual information to guide robots to cooperatively complete an insertion task based on information quality.
• The approach that the authors propose defines a constraint model that models sensor constraints explicitly and further enables indirect constraint satisfaction for alternative solutions, based on the type of the constraint.

B. Environment & Uncertainty Sampling

• The authors introduce a sampling method to incorporate environmental influence on measures of information quality.
• Then, the authors compute the influence of these samples on the measures of information quality as dictated by the application.
• Another advantage is that sensor uncertainty can be easily incorporated by sampling again on these environment samples based on the uncertainty model of the range sensor using the Metropolis-Hastings algorithm [4], which can sample from any density function using a candidate generation function.
• Based on the environment’s complexity with respect to the robots, the authors then choose a granularity (i.e., density of particles) for sample creation.
• The created samples naturally form a geometric representation of the environment.

C. Measures of Information Quality

• Since samples from the constrainer itself usually have no impact on measures of information quality, the authors need to separately consider these samples.
• For the robot tracking and navigation tasks, the authors simply assume that samples within a short distance (e.g., 0.2m) from the detected configuration of the constrainer are samples from the constrainer.
• The assumption is generally true unless geometric structures have to be specifically modeled (e.g., finding triangle shaped objects in the environment).
• Since samples from the constrainer are generally assumed to have no influence, the probability of risk for each sample si should be weighted by 1.0−ηi, where ηi is the probability of the sample being the constrainer.
• The weight for the sensor quality measurement is simply the joint probability of no risk considering all environment samples.

D. Motion Model & Motion Sampling

• The motion model is used to predict the resulting configuration given the current configuration and motion vector, Fm : (P, V ) → P .
• The authors use the common differential drive motion model, which has the form r = v/ω, where r is the radius of movement.
• Since measures of information quality can vary from application to application, methods for computing optimal solutions are generally impossible.
• Figure 1(b) explains the sampling and selection process in a simple scenario.

E. Constraint Model

• The constraint model enables indirect constraint satisfaction for alternative solutions based on the type of the constraint.
• The model is then combined with other models to compute the information quality measurements for these solutions in case of unsatisfiable constraints.
• In the robot navigation task, for successful maneuvering through a narrow hallway, constraints must be relaxed such that some follower robots switch from direct tracking of the leader to indirect tracking through other follower robots.
• For the box pushing task, the constraint for tracking the box should be relaxed by tracking the pusher robots to infer the box pushing direction when view of the box is blocked.
• Ri is initially configured to follow Rj .

IV. EXPERIMENTS & RESULTS

• The authors demonstrate their approach by applying it to two applications in simulation using Stage and with physical robots.
• In the robot tracking task, a tracking robot uses either a camera or a laser sensor to detect the target and tries to keep the target in its sight.
• The authors often assume that only one of these robots has a localization capability (e.g., using GPS) and others can detect teammates using either a camera or laser sensor.
• The goal is for all the robots to reach the goal position.
• To enhance robustness, the authors implement a simple error recovery method for going to the last seen target or leader robot position when the target or leader is lost.

A. Simulations

• The authors first compare their model for the tracking task with the specific approach of [1].
• In both environments, the robot starting from the bottom is the tracking robot.
• One 3Statistical analysis is not possible for the approach of [1] due to unavailability of the software platform on which it was run.
• To show these effects, the authors implement a multiple target tracking task with different sensor quality models.
• For all configurations, the robots reconfigure using the constraint model at different times due to occlusions and reach the goal position successfully.

B. Physical Experiments

• For the tracking and navigation tasks in physical experiments, the authors use the technique presented in [11] for constrainer detection using cameras.
• The authors ran the robot tracking task using both approaches in five different initial configurations and compare the results.
• Furthermore, the average percentage of time in track for successful runs is also much higher compared to the baseline approach.
• The authors believe that by reducing the computational load and incorporating motion prediction, the performance of their approach can be further improved.
• Experiments show distinct behaviors of the follower robots.

Figures

Table 2: Sample

Table 1.

Figure 4. View eal-Mogadishu Environment

Figure 2. A Top-down View of the Terrain for the Mogadishu Test-Bed with the Starting ocations of the Soar (blue), AI.Implant (red), and PMFserv (Green) Agents

Figure 9. Somalian Women Acting as Shields, Then Fleeing After the Militiaman is Killed

Figure 5. GSP Tree Structures for Three Archetype Agent Classes

Figure 6. Editor Windows for World Markups and Affordances

Figure 7. Somalian Male Before and After Physical Exertion

Figure 3. Architecture Adopted for Interchange and Integration of PMFserv with Unreal Tourn ament

Figure 8. Emotion Display of Looter Being Chased Away from Helicopter by Ranger (Player)

Full text

*:5B1>?5@E;2&1::?E8B-:5-*:5B1>?5@E;2&1::?E8B-:5-
(/4;8->8E;99;:?(/4;8->8E;99;:?
1<->@91:@-8&-<1>?( 1<->@91:@;281/@>5/-8(E?@19?:35:11>5:3

A9-:14-B5;>$;018?2;>31:@?5:(59A8-@;>?-:0-91?A9-:14-B5;>$;018?2;>31:@?5:(59A8-@;>?-:0-91?
&->@ -91.;@:35:11>5:3C5@4&$?1>B&->@ -91.;@:35:11>5:3C5@4&$?1>B
->>E(58B1>9-:
*:5B1>?5@E;2&1::?E8B-:5-
.-?58?1-?A<1::10A
:-:-"4->-@4E
*:5B1>?5@E;2&1::?E8B-:5-
"1B5:%>51:
*:5B1>?5@E;2&1::?E8B-:5-
!-?;:;>:C188
*:5B1>?5@E;2&1::?E8B-:5-
;88;C@45?-:0-005@5;:-8C;>7?-@4@@<?>1<;?5@;>EA<1::10A1?1,<-<1>?
&->@;2@4181/@>5/-8-:0;9<A@1>:35:11>5:3;99;:?
'1/;991:0105@-@5;:'1/;991:0105@-@5;:
->>E(58B1>9-::-:-"4->-@4E"1B5:%>51:-:0!-?;:;>:C188A9-:14-B5;>$;018?2;>
31:@?5:(59A8-@;>?-:0-91?&->@ -91.;@:35:11>5:3C5@4&$?1>B<>58
&;?@<>5:@B1>?5;:&A.85?4105:
&>1?1:/1)181;<1>-@;>?-:0+5>@A-8:B5>;:91:@?
+;8A91 ??A1<>58
<-31?
&A.85?41>*'#4@@<0D0;5;>3<>1?
)45?<-<1>5?<;?@10-@(/4;8->8E;99;:?4@@<?>1<;?5@;>EA<1::10A1?1,<-<1>?
;>9;>15:2;>9-@5;:<81-?1/;:@-/@>1<;?5@;>E<;.;DA<1::10A

A9-:14-B5;>$;018?2;>31:@?5:(59A8-@;>?-:0-91?&->@ -91.;@A9-:14-B5;>$;018?2;>31:@?5:(59A8-@;>?-:0-91?&->@ -91.;@
:35:11>5:3C5@4&$?1>B:35:11>5:3C5@4&$?1>B
.?@>-/@.?@>-/@
$-:E<>;0A/1>?-:0/;:?A91>?;2813-/E@>-5:5:3?59A8-@;>-:03-911:B5>;:91:@?->1.135::5:3@;
1:B5?5;:-:1C1>-C41>1<?E/4;?;/5;<4E?5;8;35/9;018?/;A80.15:@1>;<1>-@10@;1:4-:/1@415>
1:B5>;:91:@??59A8-@5;:;24A9-:-31:@?)45?<-<1>1D<8;>1?C41@41>C1/;A8019.10;A>.14-B5;>
9;0185:32>-91C;>701?/>5.105:@41/;9<-:5;:<-<1>&->@.145:0-813-/EF>?@<1>?;:?4;;@1>
3-911:B5>;:91:@@;>1/>1-@1<;>@5;:?;2@418-/7-C7;C:?/1:->5;(1/@5;:-9<85F1?@41
5:@1>;<1>-.585@E:110?-:0/4-881:31?/;:2>;:@5:3@41F180<>1?1:@?@41=A1?@5;:?@4-@->11D-95:10-:0
01?/>5.1?@41@1?@?/1:->5;(1/@5;:?-:0>1B51C@41?;2@C->1-:07:;C810311:35:11>5:3
91@4;0;8;3E>1?<1/@5B18E:11010@;/>1-@1@41?E?@19-:0<;<A8-@15@C5@4.;@?'1?A8@?(1/@5;:-:0
05?/A??5;:(1/@5;:>1B1-8@4-@C1C1>1-.81@;31:1>-@1<8-A?5.81-:0-0-<@5B1>1/>1-@5;:?;2
(;9-85-:/>;C0?9585@5-C;91:-/@5:3-??45180??A5/501.;9.1>?-:09;>18?;@41>1->1?<1/5F/
81??;:?81->:10-.;A@C-E?@;-0B-:/1@41F180?;@4-@?A/45:@1>;<1>-.585@51?C588.1/;919;>1
-22;>0-.81-:0C501?<>1-0
"1EC;>0?"1EC;>0?
4A9-:.14-B5;>9;018?/A8@A>1-:019;@5;:??59A8-@;>-:0-31:@5:@1>;<1>-.585@E/;9<;?-.585@E
5?/5<85:1?5?/5<85:1?
81/@>5/-8-:0;9<A@1>:35:11>5:3
;991:@?;991:@?
&;?@<>5:@B1>?5;:&A.85?4105:
&>1?1:/1)181;<1>-@;>?-:0+5>@A-8:B5>;:91:@?
+;8A91 ??A1
<>58<-31?
&A.85?41>*'#4@@<0D0;5;>3<>1?
)45?6;A>:-8->@5/815?-B-58-.81-@(/4;8->8E;99;:?4@@<?>1<;?5@;>EA<1::10A1?1,<-<1>?

Human Behavior Models for Agents in Simulators and Games:
Part II – Gamebot Engineering with PMFserv
Barry G. Silverman, Ph.D., Gnana Bharathy, Kevin O’Brien, Jason Cornwell
Ackoff Center for Advancement of Systems Approaches (ACASA), Department of
Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA
19104-6315, USA. e-mail: barryg@seas.upenn.edu
ABSTRACT
Many producers and consumers of legacy training simulator and game environments are
beginning to envision a new era where psych-socio-physiologic models could be inter-
operated to enhance their environments' simulation of human agents. This article
explores whether we could embed our behavior modeling framework (described in Part I)
behind a legacy first person shooter 3-D game environment to recreate portions of the
Black Hawk Down scenario. Section One amplifies on the inter-operability needs and
challenges confronting the field, presents the questions that are examined, and describes
the test scenario. Sections 2 and 3 review the software and knowledge engineering
methodology, respectively, needed to create the system and populate it with bots. Results
(Section 4) and discussion (Section 5) reveal that we were able to generate plausible and
adaptive recreations of Somalian crowds, militia, women acting as shields, suicide
bombers, and more. Also, there are specific lessons learned about ways to advance the
field so that such inter-operabilities will become more affordable and widespread.
Keywords: human behavior models; culture and emotions; simulator and agent
interoperability; composability
1) Introduction
Today’s world is on the verge of an era of ubiquitous agents – autonomous characters
that assist in all endeavors at work, at home, online, in games, and in social settings. Yet
today’s agents are too easily perceived as mechanistic automatons, causing users to
experience frustration, inappropriate expectations, and/or failures of engagement and
training. Reliable pathways for creating more realistic and believable agents could
ultimately help reduce barriers to interacting with as well as to creating behaviors of
empathetic avatars, electronic training world opponents and allies, digital cast extras,
wizard helper agents, and so on.
This is no where more apparent than in the military modeling and simulation community
which is demanding human behavior models (HBMs) to satisfy a wide and expanding
range of scenario concerns. Their interest goes beyond mission-oriented military
behaviors, to also include simulations of the effects that an array of alternative
diplomatic, intelligence, military, and economic (DIME) actions might have upon the
political, military, economic, social, informational (psyops), and infrastructure (PMESII)

dimensions of a foreign region. The goal is to defeat adaptive foes adept at using local
PMESII effects to their own advantage: e.g., see Runals (2004).
If the military is to have realistic and reliable models of the effects of DIME type
operations upon PMESII dimensions, one must find ways to integrate scientific know-
how across many disciplines. As the top of Figure 1 shows, science tends to be reductive,
specialized, and siloed. Labs that study sleep deprivation don’t also study impacts of non-
lethal crowd control methods, and those specialists know little about political coalition
dynamics. Yet, each of these, and more disciplines have something of value to contribute
if we are to realistically model the type of effects just described.
Part I of this article presented a unified architecture for human behavior modeling that
seeks to straddle and synthesize models and principles from physiology/stress,
personality/culture/emotion, social/political, and cognition and perception. This is an
approach to help modelers cull scientific models and first principles from the behavioral
literatures so they can be edited, tested for their validity, and used to improve realism of
agent behavior. Obviously, many efforts such as this effort are needed to make progress.
Science continually must go through periods of synthesis across disciplines in order to
uncover its shortcomings and to regenerate. This is the feedback loop that the right side
of Figure 1 shows from synthesis to further empiric and reductive investigations. The
current push for better models is uncovering and fueling many such studies at present. It
is thus a productive time to examine synthesis of HBMs and methods for doing so.
Our computer implementation of the unified behavior architecture, PMFserv, provides
one starting synthesis of models and principles. The current article, Part II, serves as an
existence proof that this implementation can be harnessed and used to enhance agent
realism and to help model and simulate certain pre-, during, and post-conflict situations
in other cultures. Since this is a case study, the answers we uncover will be largely
limited to one instance, and not generalizable without further investigation. Also, no one
HBM is sufficient to address all the concerns, so the bottom of Figure 1 also lays out a
methodology in four boxes that raises the idea of federating other models as well. This
vision leads to three sets of questions we explore in this paper:
1) Are models drawn from the literature useful and usable as agent minds? To what
degree will they elevate an automaton into a realistic agent? Under what
conditions do these models help agents pass (fail) correspondence tests?
2) Is the legacy simulator community (military and entertainment) ready and able to
accept such plug-in models for updating the minds of bots that already exist in
their software? If not, what obstacles exist and what fixes appear warranted?
3) What is needed to improve the composability situation so that digital casts can be
created? From a knowledge engineering perspective, how do various methods and
approaches impact affordability?
The motivation behind these questions is to explore if it is reasonable to federate models
to foster composability. There is study after study that shows the lack of credible
behavioral capability of the legacy systems (e.g., see Pew & Mavor, 1998; Anon., 1995;
Bjorkman & Blemberg, 2001, among others). A federation approach could help to
preserve the investment in legacy simulator and game environments, while making newer
2

character simulations and behavioral model innovations available. This path has been
advocated by the Department of Defense, among others, who has identified a need for
interoperability of human behavior models to help improve the realism of agents in
legacy simulators: (e.g., see Finerman et al., 2001; Bjorkman, Barry, & Tyler, 2001; Toth
et al., 2003).
igure 1. The Four Stage Synthesis Methodology and How It Integrates New Science
he four numbered blocks in the Synthesis portion of Figure 1 represent a four stage
ario
Fserv
ario
s part of the case study, the client also requested that we attempt to embed the PMFserv
is
n
PEDAGOGY
PLACE PEOPLE
PLOT vs.
PLAY
5P
PEDAGOGY
PLACE PEOPLE
PLOT vs.
PLAY
5P
Available Science
•Specialty silos: reduction
•Prevailing theories/models
•1
st
principle model specs
•Field data sets
Gaps in Science
•Models missing parts
•Interdiscipline needs
•Field data needs
Science In Use: Synthesis Stages
Scientific Shifts
•Silo broadening
•New hypotheses
•Empirical studies
1.Scenario Composition
4.Model Usage
•Validity Tests
•Training & AAR
•What-If Analyses
•DIME-PMESII
•Discovery (EBO)
Scenario Engineering Application Engineering Sim Experiments
2.
Legacy
Simul-
ators &
Games
Scientific Method: Reduction
3.Model Authoring
PMFserv Modules:
•Cull Avail Science
•Structure Models
•Collect Evidence
•Assess Parameters
•Visually Program
•Test & Tune
Biology/Stress, Personality/Culture/Emotion, Social/Political, Cognition/Perception
F
and Legacy Software into Human Behavior Modeling
T
methodology that we have evolved through several studies and that is the organizing
framework of this paper. Frequently, the client has only a top level notion of the scen
to be engineered. For example in this case study, in the summer of 2002, the
DOD/Defense Modeling & Simulation Office (DMSO) wanted to see if our PM
agent behavior framework could successfully run the local crowds and militia of a
recreation of the Black Hawk Down scenario. To help the client develop their scen
further, we use a process labeled 5P (1
st
stage in Figure 1) and explained more fully in
Section 1.1.
A
agent minds behind a pre-existing simulator. This is question set 2 above, and it is the
nature of HBM today that one often must embed behind a client’s legacy simulator. Th
2
nd
stage of the methodology is a challenge. In a recent survey of five legacy combat
simulators (JSAF, ModSAF, OneSAF, DISAF, JCATS), it was found that (1) one ofte
can’t discover if a given behavior exists or what level of fidelity its modeled at; (2) the
software is growing constantly; (3) verification and validation needs of the legacy
3

Frequently asked questions

Q1. What are the contributions in "Human behavior models for agents in simulators and games: part ii gamebot engineering with pmfserv" ?

This paper explores whether the authors could embed their behavior modeling framework ( described in the companion paper, Part 1 ) behind a legacy first person shooter 3D game environment to recreate portions of the Black Hawk Down scenario. Section 1 amplifies the interoperability needs and challenges confronting the field, presents the questions that are examined, and describes the test scenario. 

Q2. What have the authors stated for future works in "Human behavior models for agents in simulators and games: part ii gamebot engineering with pmfserv" ?

Most simulation developers and sponsors are now working to extend their systems to permit interchange with other approaches and other vendors. The authors hope that their research will help contribute to that advance, as summarized in this two-part article. First, a sea change will arise in the field of behavioral modeling, which will shift from reductive, silo-separated specialties, to a proliferation of collaborating best-of-breed PMFs, AI systems, and A-life components created by and widely shared amongst distributed researchers. Second, there will be few technological barriers to entry for crafting purposive behaviors of avatars, allies, crowds, opponents, digital cast extras, etc. 

Q3. What tests should be supported to fully understand and trust agent behavior models?

To fully understand and trust agent behavior models, a number of validation tests should be supported such as individual PMF tests, further correspondence tests, Turing tests, and competing agent model tests, among others. 

Q4. How many lines of code did the custom unreal script need to be created?

Since there are no naming conventions or translation standards in general for human behavior models, the resulting Custom Unreal Script was difficult to create and grew to about 1,000 lines of code, code that is not itself very reusable. 

Q5. What did the bots have to do to be able to perform the low level behaviors?

These shareware bots existed with many of the low level behaviors including breathing, a celebratory animation that looks a bit like break dancing, running, picking up a weapon, shooting, dying, and the like. 

Q6. What is the precursor to this effort?

As with their 6-step process, the precursor to this effort is to fill out spreadsheets on the markups for each object from each obs perspective. 

Q7. How many students have used this approach to produce term papers?

Over the past few years there have been a dozen student projects th successfully used this spreadsheet approach to produce term papers that cull referen from the literature that support the various tree branches and weights assigned to bots of given archetype and affordance levels for various world objects. 

Q8. What is the effect of interoperation on the validity of a PMFserv agent?

Since most behavior observed in PMFserv agents is the result of many subsystems and PMFs interoperating, this alters the validity of any given PMF at runtime. 

Q9. What is the common editor in PMFserv?

To help with all that, PMFserv includes a number of editors including bot and object creation editor, affordance editor, action editor, and others. 

Q10. What type of structure did the authors use for the goal tree?

as Figure 5 also shows, the authors often used a Maslow type of structure for short term needs in the Goal tree, particularly for f and cell members.