Inferring colocation and conversation networks from privacy-sensitive audio with implications for computational social science

TL;DR: New methods for inferring colocation and conversation networks from privacy-sensitive audio are applied in a study of face-to-face interactions among 24 students in a graduate school cohort during an academic year, and show that networks derived from colocations and conversation inferences are quite different.

Abstract: New technologies have made it possible to collect information about social networks as they are acted and observed in the wild, instead of as they are reported in retrospective surveys. These technologies offer opportunities to address many new research questions: How can meaningful information about social interaction be extracted from automatically recorded raw data on human behaviorq What can we learn about social networks from such fine-grained behavioral dataq And how can all of this be done while protecting privacyq With the goal of addressing these questions, this article presents new methods for inferring colocation and conversation networks from privacy-sensitive audio. These methods are applied in a study of face-to-face interactions among 24 students in a graduate school cohort during an academic year. The resulting analysis shows that networks derived from colocation and conversation inferences are quite different. This distinction can inform future research in computational social science, especially work that only measures colocation or employs colocation data as a proxy for conversation networks.

Summary

1. INTRODUCTION

• The automated recording of real-world speech is crucial because, despite the rise in on-line interactions, face-to-face communication is still people’s primary mode of social interaction [Baym et al. 2004].
• This requirement gives rise to two problems.
• Ideally, a privacysensitive recording technique will process incoming audio in order to discard any information deemed too invasive while still preserving data useful for sociological inquiry.

2. PRIVACY-SENSITIVE CONVERSATION MODELING

• When collecting situated conversation data it is necessary to protect the privacy of not just people who willingly consent to wear a recording device, but also of those who may come within range of the microphones.
• For this purpose, destructive processing of the audio should yield a feature set that prevents us from reconstructing intelligible speech or inferring the identities of anyone not wearing a device.
• At the same time, the features must still contain enough information to allow conversations to be found and meaningful inferences made about those ACM Transactions on Intelligent Systems and Technology, Vol. 2, No. 1, Article 7, Pub. date: January 2011. conversations.
• Energy can reveal a person or group’s interest in the conversation [GaticaPerez et al. 2005].
• The method proposed by Corman and Scott [1994] computes normalized cross-correlation between raw audio signals and concludes that two people are in a conversation if their correlation coefficients are above a threshold estimated from labeled data.

2.1 Privacy-Sensitive Features

• Following Basu [2002], their approach to extracting non-linguistic speech information builds on methods for detecting voiced human speech.
• In a spectrogram, time runs along the x-axis and frequencies increase along the y-axis; color indicates energy at a given frequency.
• The harmonicity in the spectrogram shows that voiced speech has a low spectral entropy, compared to non-voiced regions.
• Narrow spectrum noise can also create strong autocorrelation peaks.
• The precise procedure for computing features is as follows:.

2.2 Extracting Conversation Data

• To gather data about face-to-face conversations, the authors ask multiple people to wear recording devices each of which saves separate streams of the privacy-sensitive features described above.
• Finally, once colocated groups and speakers have been identified, the authors can conclude that people who are colocated and speaking are in conversation together and then extract further features of their conversation (Section 2.3).
• For each recorded stream, the authors use the forward-backward algorithm [Rabiner 1989] to infer p(V ta|xa): the posterior probability of voiced speech in each frame, given the entire recorded stream.
• Successful colocation detection requires clustering together segments of data from miked individuals when they are in a conversation.
• There is a sharp peak of high mutual information values ACM Transactions on Intelligent Systems and Technology, Vol. 2, No. 1, Article 7, Pub. date: January 2011.

2.3 Conversation Data

• The steps described so far provide ways of determining who is physically colocated with whom and who is speaking when, but they do not provide a method for determining who is in conversation with whom.
• Thus, an evaluation comparing the resulting inferred conversations to the “in conversation with” ground truth label yields exactly the same results as comparing their voicing-based colocation inference to the “in conversation with” ground truth.
• The table of those results is identical to Table II, and thus the authors omit it for space.

3. THE SPOKEN NETWORKS CORPUS

• Using the conversation detection methods from the previous section, the authors collected a corpus of real-world face-to-face conversations among 24 research subjects over an extended period of time.
• This section first contrasts their effort with earlier data collection projects (Section 3.1), it then explains the procedure used to gather the data (Section 3.2), provides summary statistics about the data (Section 3.3), and shows novel measures of social behavior that can be easily extracted form the data (Section 3.4).

3.1 Related Work

• The authors project integrates two distinct streams of research.
• Second, it monitors these interactions over an entire academic year, building on previous work in collecting temporal social network data.
• Existing efforts at collecting real-world speech data have considered settings—meetings, phone conversations, interviews [Ang 2002; McCowan et al.
• The most relevant and groundbreaking real-world social behavior data collection (and the immediate ancestor of this work) has used the sociometer: a wearable platform combining infrared, motion, and—most importantly—audio sensors [Choudhury and Pentland 2003].
• Methods employing audio data have many advantages.

3.2 Data Collection Method

• The data collection effort presented in this work descends from the original sociometer study, but differs in the research context and design.
• The subjects recorded data everywhere they went, indoors and outdoors: class, lunch, study groups, meetings, spontaneous social gatherings, etc. Data was saved to a 2-GB Secure Digital (SD) flash memory card on the PDA.
• Research subjects completed questionnaires before beginning the school year, at the end of each data collection week, and following the end of the school year.
• All conversation data discussed here was collected using the same platform: an HP iPAQ hx4700 PDA with an attached multi-sensor board containing eight different sensors.
• Because all of the PDA’s software and settings are stored in volatile RAM and are completely lost if the battery fully discharges, this led to many Monday mornings of lost recording time while PDAs were reconfigured.

3.3 Collected Data

• Figure 8 shows beanplots [Kampstra 2008] of the average number of hours collected per day for each collection week.
• The first three weeks (i.e., representing the first three months of the academic year) show an increase in the amount of data collected as the subjects initially learned how to use the devices and the authors resolved battery and software problems as previously described.
• Recording hours diminished slightly in the later weeks, also due partly to technical problems with the cables and perhaps because the participants became fatigued or the study became less novel to them.
• While there is no moment when all subjects are recording (the maximum number of simultaneous recordings is 21), there is enough overlap in the data for it to contain many interactions among their subjects.

3.4 Basic Behavioral Inferences

• Data processing follows the three steps described in Section 2: colocation detection, speaker segmentation, and conversation extraction.
• Recall from ACM Transactions on Intelligent Systems and Technology, Vol. 2, No. 1, Article 7, Pub. date: January 2011.
• During this academic term, most subjects attended a class that met from 10:30 am to 12:00 pm on these days, so many students arrived at school and began recording before that class.
• Figure 11 shows the inferences for colocation using both energy and voicing mutual information, as well as the conversation grouping.
• Because of that, when few people are recording even a small group of interacting subjects will appear as a larger proportion in the plot.

3.5 Basic Network Analyses

• Constructing networks from survey data is usually simple: they are often just the union of self-reported ties for each actor in the network.
• As with the edge value distributions, the values for the colocation degrees are much higher than those for conversation degrees and the two kinds networks seem to be very different with regard to degree.
• As with the clustering coefficient, the triangle count does not generalize as easily to weighted networks as degree and density, but, following Saramäki et al. [2007], the authors can define a weighted triangle value as Tijk = (YijYikY jk)1/3 (19).
• That difference may explain the bimodal colocation degree distributions of weeks 4 through 7 ), where there seems to be a distinction between pairs who spend much time together and pairs that only come together in passing.

4. CONCLUSION

• The authors have outlined a set of privacy-sensitive features that can be computed from incoming audio data in real-time.
• The authors have shown how to use those features to determine who was physically colocated with whom, both at the granularity of a room in a building and at the more elastic “acoustic proximity” needed to have a face-to-face conversation.
• The authors have used the privacy sensitive features to infer who was speaking when, and combined those inferences with colocation inference to determine who was in conversation with whom.
• This conversation detection can handle conversations with any number of participants, extending beyond previous methods that were limited to dyadic conversations only.
• The authors constructed weighted networks of social behavior and examined basic descriptive statistics in order to compare social networks defined by colocation events to networks defined by conversation events.

Figures

Fig. 6. The MSB. Microphone is at top.

Fig. 2. Audio of a male voice saying “University of Washington Spoken Networks.”

Fig. 5. Histogram of voicing mutual information values for one week of data with fitted mixture model.

Fig. 8. Average hours recorded per day for each subject in each week. Black lines are data points: the average for one person for that week. Blue “beans” are kernel density estimates. Green lines are medians and red lines are means.

Fig. 9. Number of people simultaneously recording each 20 second window with at least one person recording. Stacked blue boxes are histograms with one bin for each possible number of simultaneous recordings. The width of the box reflects the number of windows simultaneously recorded by the corresponding number of subjects. Green lines are medians and red lines are means.

Table IV. Other Colocation Techniques Compared to True Conversation Grouping

Fig. 12. Conversation networks for each week. Edge shades correspond to proportion of time spent in conversation.

Fig. 4. Inferred voicing posterior (blue line, right y axis) overlayed on examples from Figure 2.

Table III. Other Colocation Techniques Compared to True Room-Level Colocation

Fig. 14. Degree distributions.

Fig. 15. Weighted clustering coefficient distributions.

Table V. Raw Speaker Pseudo-Confusion Matrix

Table VI. Raw Speaker Segmentation Performance

Fig. 7. The data collection kit worn by each subject.

Fig. 11. Proportion of recording pairs that are physically colocated in accordance with energy correlation, voicing mutual information, and in conversation.

Table II. Colocation Inference Compared to True Conversation Grouping

Fig. 1. Conceptual schematic of the source-filter model.

Fig. 16. Weighted triangle value distributions.

Table VIII. Smoothed Speaker Segmentation Performance

Table VII. Smoothed Speaker Pseudo-Confusion Matrix

Table I. Colocation Inference Compared to True Room-Level Colocation

Full text

7
Inferring Colocation and Conversation
Networks from Privacy-Sensitive Audio
with Implications for Computational
Social Science
DANNY WYATT
University of Washington
TANZEEM CHOUDHURY
Dartmouth College
JEFF BILMES
University of Washington
and
JAMES A. KITTS
Columbia University
New technologies have made it possible to collect information about social networks as they are
acted and observed in the wild, instead of as they are reported in retrospective surveys. These
technologies offer opportunities to address many new research questions: How can meaningful
information about social interaction be extracted from automatically recorded raw data on human
behavior? What can we learn about social networks from such fine-grained behavioral data? And
how can all of this be done while protecting privacy? With the goal of addressing these questions,
this article presents new methods for inferring colocation and conversation networks from privacy-
sensitive audio. These methods are applied in a study of face-to-face interactions among 24 students
in a graduate school cohort during an academic year. The resulting analysis shows that networks
derived from colocation and conversation inferences are quite different. This distinction can inform
future research in computational social science, especially work that only measures colocation or
employs colocation data as a proxy for conversation networks.
This work was supported by NSF grants IIS-0433637 and IIS-0845683.
Authors’ addresses: D. Wyatt, University of Washington, Department of Computer Science and
Engineering, Box 352350, Seattle, WA 98195-2350; email: danny@cs.washington.edu; T. Choud-
hury, Dartmouth College, 6211 Sudikoff Lab, Hanover, NH 03755; email: tanzeem.choudhury@
dartmouth.edu; J. A. Bilmes, University of Washington, Seallte, Department of Electrical Engi-
neering, Box 352500, Seattle, WA 98195-2500; email: bilmes@u.washington.edu; James A. Kitts,
Columbia University, Graduate School of Business, 704 Uris Hall, 3022 Broadway, New York, NY
10027; email: jak2190@columbia.edu.
Permission to make digital or hard copies of part or all of this work for personal or classroom use
is granted without fee provided that copies are not made or distributed for profit or commercial
advantage and that copies s how this notice on the first page or initial screen of a display along
with the full citation. Copyrights for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers,
to redistribute to lists, or to use any component of this work in other works requires prior specific
permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn
Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or permissions@acm.org.
C

2011 ACM 2157-6904/2011/01-ART7 $10.00
DOI 10.1145/1889681.1889688 http://doi.acm.org/10.1145/1889681.1889688
ACM Transactions on Intelligent Systems and Technology, Vol. 2, No. 1, Article 7, Pub. date: January 2011.

7:2
•
D. Wyatt et al.
Categories and Subject Descriptors: G.3 [Probability and Statistics]: Probabilistic Algorithms;
H.5.5 [Information Interfaces and Presentation]: Sound and Music Computing
General Terms: Algorithms, Experimentation, Human Factors, Security
Additional Key Words and Phrases: Social networks, mobile sensing
ACM Reference Format:
Wyatt, D., Choudhury, T., Bilmes, J., and Kitts, J. A. 2011. Inferring colocation and conversation
networks from privacy-sensitive audio with implications for computational social science. ACM
Trans. Intell. Syst. Technol. 2, 1, Article 7 (January 2011), 41 pages.
DOI = 10.1145/1889681.1889688 http://doi.acm.org/10.1145/1889681.1889688
1. INTRODUCTION
Much social network research has relied on data collected via surveys that ask
subjects to report their social ties (e.g., Goodreau et al. [2009]) or recall their pre-
vious social interactions (e.g., Lazega and van Duijn [1997]). When self-reports
of recalled interactions have been compared to independent observations, how-
ever, the reliability of subjects’ answers has been shockingly poor [Killworth
and Bernard 1976; Bernard and Killworth 1977, 1979; Bernard et al. 1980,
1982]. An early study came to the dire conclusion that “people do not know, with
any accuracy, those with whom they communicate” [Bernard and Killworth
1977]. Later studies found that durable, long-term patterns of communica-
tion are reliably reported, but moment-to-moment social interactions are not
[Freeman et al. 1987]. More troubling for research into network structure, in-
dividuals tend to “fill in” non-existent interactions if they would increase the
transitivity of the network [Freeman 1992]. Faced with these challenges, some
researchers lamented that “unfortunately, most naturally occurring interactive
behavior (the stuff of which networks are built) is neither observable nor con-
veniently recorded in some automated fashion” [Killworth and Bernard 1979].
That statement is no longer true. New technologies have made it possible
to collect information about social behavior as it is enacted, instead of as it is
recalled after-the-fact. Phone calls, text messages, emails, instant messages,
on-line chat sessions, social media posts, and any other kind of electronically
mediated communication can be automatically recorded for large groups of
people, over long periods of time. Portable audio recording devices have grown
in capacity while becoming smaller, cheaper, and more powerful, making it
easier to record face-to-face conversations. In fact, wearable sensors now allow
us to automatically record natural and spontaneous speech for an entire group
of people over a long period of time.
The automated recording of real-world speech is crucial because, despite the
rise in on-line interactions, face-to-face communication is still people’s primary
mode of social interaction [Baym et al. 2004]. Unlike methods previously em-
ployed for speech data derived from laboratory contexts, our proposed method
would capture truly spontaneous speech that arises in situ as people enact their
actual, lived relationships. For that reason, we refer to such data as situated
speech data—data gathered in the wild—to contrast it with other speech data
recorded in constrained or contrived settings.
ACM Transactions on Intelligent Systems and Technology, Vol. 2, No. 1, Article 7, Pub. date: January 2011.

Inferring Colocation and Conversation Networks from Privacy-Sensitive Audio
•
7:3
Of course, obstacles to gathering situated spontaneous speech still remain,
especially concerns about privacy. To capture truly natural interactions while
providing a full picture of a social network, we must record people as they
freely go about their lives. This requirement gives rise to two problems. First,
uninvolved parties could be recorded without their consent—a scenario that,
if raw audio is involved, is always unethical and often illegal. Second, people
may change their behavior if they know they are being recorded. For both
of those reasons, a level of privacy must be maintained. Ideally, a privacy-
sensitive recording technique will process incoming audio in order to discard
any information deemed too invasive while still preserving data useful for
sociological inquiry.
This dilemma illuminates what is perhaps a fundamental trade-off between
privacy and quality when automatically recording social behavior. Subjects are
unlikely to consent to large-scale, unrestricted recordings of their behavior,
so some sociologically useful information must almost surely be destroyed.
Therefore, set of features that allow us to balance this trade-off between privacy
and quality is needed.
In this article, we present exactly such a set of privacy-sensitive features,
together with a method for using this feature set to find colocation and conver-
sation events in separately recorded streams of audio data. In evaluation using
non-privacy-preserving test data—where access to ground truth is possible—
our method performs better than earlier methods.
We then use our proposed method to derive networks of colocation and face-
to-face conversation among 24 graduate students over the course of an academic
year. Networks created from colocations and conversations appear to be quite
different—a result that can impact and inform future research in computa-
tional social science.
The remainder of this article is divided into two broad sections. Section 2
presents the methods for discovering physically colocated and conversing peo-
ple from privacy-sensitive audio data, then assesses the performance of these
methods. Section 3 covers the Spoken Networks project, a data collection effort
that employed the proposed methods to study a real-world network over an
extended period of time, demonstrating new insights that may be available
through these lenses.
2. PRIVACY-SENSITIVE CONVERSATION MODELING
When collecting situated conversation data it is necessary to protect the privacy
of not just people who willingly consent to wear a recording device, but also
of those who may come within range of the microphones. For this purpose,
destructive processing of the audio should yield a feature set that prevents us
from reconstructing intelligible speech or inferring the identities of anyone not
wearing a device. A further constraint on the feature set is that all features
must be computed in real-time within the limited computational resources of
a wearable device—no raw audio should ever be stored, even temporarily.
At the same time, the features must still contain enough information to
allow conversations to be found and meaningful inferences made about those
ACM Transactions on Intelligent Systems and Technology, Vol. 2, No. 1, Article 7, Pub. date: January 2011.

7:4
•
D. Wyatt et al.
0 500 1000 1500 2000 2500 30000 500 1000 1500 2000 2500 30000 500 1000 1500 2000 2500 3000
Frequency (Hz) Frequency (Hz) Frequency (Hz)
Source Filter Speech
Fig. 1. Conceptual schematic of the source-filter model.
conversations. Fortunately, the nonlinguistic aspects of a conversation—who
speaks when and for how long, how loud, and at what pitch—still allow for
many useful analyses. Interruptions and speaking time reveal information
about status and dominance [Hawkins 1991]. Speaking rate reveals informa-
tion about a speaker’s level of mental activity [Hurlburt et al. 2002]. Energy
(loudness) can reveal a person or group’s interest in the conversation [Gatica-
Perez et al. 2005]. Pitch may be used for inferring emotion [Dellaert et al. 1996],
and energy and duration of voiced and unvoiced regions are also informative
emotional features [Schuller et al. 2004].
Here we present a set of privacy-sensitive features that can be extracted
from an audio stream in real-time (Section 2.1), along with methods for using
those features to automatically determine who is in conversation with whom
(Section 2.2) and how people are speaking (Section 2.3).
Related Work. To the best of our knowledge, prior to this research, there
were only two existing methods for finding conversations in separately recorded
streams of audio. The method proposed by Corman and Scott [1994] computes
normalized cross-correlation between raw audio signals and concludes that
two people are in a conversation if their correlation coefficients are above a
threshold estimated from labeled data. Obviously, using raw audio does not
protect privacy, but a privacy-sensitive variant of their method is considered
below. Similarly, the method proposed by Basu [2002] computes t he mutual
information between binary signals that represent voiced/unvoiced speech and
places two people in a conversation if their mutual information is above a
pre-specified threshold. Our work extends Basu’s method in three important
ways: (i) to detect multiperson conversations (not just dyadic), (ii) to operate at
a finer time granularity while still producing a “smooth” inference over time,
and (iii) to learn its threshold in an unsupervised manner.
2.1 Privacy-Sensitive Features
Following Basu [2002], our approach to extracting non-linguistic speech infor-
mation builds on methods for detecting voiced human speech. A basic model for
the production of human speech is the standard source-filter model [Quatieri
2001] shown in Figure 1. As its name suggests, the source-filter model posits
two semi-independent components of speech production: (1) a source sound
that is generated by the glottis and passed through (2) the filter, realized by
the vocal tract, that shapes the spectrum of the source.
ACM Transactions on Intelligent Systems and Technology, Vol. 2, No. 1, Article 7, Pub. date: January 2011.

Inferring Colocation and Conversation Networks from Privacy-Sensitive Audio
•
7:5
The source can be voiced or unvoiced. If it is voiced, the vocal cords are
vibrating at what i s called the fundamental frequency, or F0, which constitutes
the pitch at which the person is speaking. A sequence of speech will alternate
rapidly between voiced and unvoiced segments. Prosodic features of speech—
intonation, stress, duration—are described by how the fundamental frequency
and energy (volume) change during speech.
The source sound is shaped into words by changing the shape of the vocal
tract. It is the frequency response of the vocal tract, particularly the resonant
peaks known as formants, that contains information about the phonemes that
are constituent parts of spoken words. Any processing of the audio that removes
information about the formants will ensure that intelligible speech cannot be
synthesized from the signal that remains.
Thus, to find conversations and retain information about how people are
speaking, we save information about the source while discarding (almost) all
information about the filter. We argue below that this preserves sociologically
useful information.
The first step in that process is finding voiced speech. Figure 2(a) shows the
spectrogram for a male voice saying the phrase “University of Washington Spo-
ken Networks.” In a spectrogram, time runs along the x-axis and frequencies
increase along the y-axis; color indicates energy at a given frequency. In this ex-
ample, all of the phonemes are voiced except those for “s,” “t,” “sh,” “p,” and “k.”
The strong harmonics are indicators of voiced speech and we take advantage
of that harmonicity to find segments of voiced speech.
Three features that have been shown to be useful for robustly detecting
voiced speech under varying noise conditions are: (i) noninitial maximum auto-
correlation peak, (ii) the total number of autocorrelation peaks, and (iii) relative
spectral entropy [Basu 2002]. To provide an intuition for the first two features,
Figure 2(b) shows the autocorrelogram for the example phrase. As in the spec-
trogram, time runs along the x-axis. The y-axis shows increasing lags at which
the autocorrelation is computed, and colors show the value of the autocorrela-
tion. The voiced segments show fewer, stronger peaks. All three features are
shown in Figure 2(c). During voiced segments, the number of autocorrelation
peaks drops, while the maximum autocorrelation value and relative spectral
entropy rise.
The harmonicity in the spectrogram shows that voiced speech has a low
spectral entropy, compared to non-voiced regions. However, in many environ-
ments there can be noise centered strongly at a specific frequency. Figure 2(a)
shows two possible examples of such noise: a low frequency hum (from 300
to 500 Hz) that may be an air conditioner, and a sharp high frequency noise
(around 6400 Hz) that is probably a computer fan or hard drive. Such narrow
spectrum noise will lower the general environmental spectral entropy. The rel-
ative s pectral entropy is the relative entropy (also known as Kullback-Leibler
divergence, see Eq. (2)) between an instantaneous normalized spectrum and
the mean normalized spectrum for a much longer window of time. Relative
spectral entropy captures the quick change in entropy caused by short seg-
ments of voiced speech while smoothing away any environmental reductions in
entropy. Narrow spectrum noise can also create strong autocorrelation peaks.
ACM Transactions on Intelligent Systems and Technology, Vol. 2, No. 1, Article 7, Pub. date: January 2011.

Frequently asked questions

Q1. What are the contributions in "Inferring colocation and conversation networks from privacy-sensitive audio with implications for computational social science" ?

New technologies have made it possible to collect information about social networks as they are acted and observed in the wild, instead of as they are reported in retrospective surveys. With the goal of addressing these questions, this article presents new methods for inferring colocation and conversation networks from privacysensitive audio. These methods are applied in a study of face-to-face interactions among 24 students in a graduate school cohort during an academic year. This distinction can inform future research in computational social science, especially work that only measures colocation or employs colocation data as a proxy for conversation networks.