Social data in digital form—including user-generated content, expressed or implicit relations between people, and behavioral traces—are at the core of popular applications and platforms, driving the research agenda of many researchers. The promises of social data are many, including understanding “what the world thinks” about a social issue, brand, celebrity, or other entity, as well as enabling better decision-making in a variety of fields including public policy, healthcare, and economics. Many academics and practitioners have warned against the naive usage of social data. There are biases and inaccuracies occurring at the source of the data, but also introduced during processing. There are methodological limitations and pitfalls, as well as ethical boundaries and unexpected consequences that are often overlooked. This paper recognizes the rigor with which these issues are addressed by different researchers varies across a wide range. We identify a variety of menaces in the practices around social data use, and organize them in a framework that helps to identify them.

“For your own sanity, you have to remember that not all problems can be solved. Not all problems can be solved, but all problems can be illuminated.” –Ursula Franklin1

/pdf/social-data-biases-methodological-pitfalls-and-ethical-44fpjwmjjy.pdf

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries.

Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions.Learn how toUse the scientific method to evaluate hypotheses using controlled experiments Define key metrics and ideally an Overall Evaluation CriterionTest for trustworthiness of the results and alert experimenters to violated assumptionsBuild a scalable platform that lowers the marginal cost of experiments close to zeroAvoid pitfalls like carryover effects and Twyman's lawUnderstand how statistical issues play out in practice.

Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

In this paper, we develop a taxonomy of conceptual models and proxy data used for digital analysis of human emotional expression and outline how the combinations and permutations of these models and data impact their incorporation into artificial intelligence (AI) systems. We argue we should not take computer scientists at their word that the paradigms for human emotions they have developed internally and adapted from other disciplines can produce ground truth about human emotions; instead, we ask how different conceptualizations of what emotions are, and how they can be sensed, measured and transformed into data, shape the ethical and social implications of these AI systems.

https://dl.acm.org/doi/pdf/10.1145/3442188.3445939

The Ethics of Emotion in Artificial Intelligence Systems

The purpose of consent is to let a person waive her rights of control over some aspects of her life. But becoming part of a research study does not always seriously undermine one's control.

When and Why Is Research without Consent Permissible

Practicality was a valued attribute of academic psychological theory during its initial decades, but usefulness has since faded in importance to the field. Theories are now evaluated mainly on their ability to account for decontextualized laboratory data and not their ability to help solve societal problems. With laudable exceptions in the clinical, intergroup, and health domains, most psychological theories have little relevance to people's everyday lives, poor accessibility to policymakers, or even applicability to the work of other academics who are better positioned to translate the theories to the practical realm. We refer to the lack of relevance, accessibility, and applicability of psychological theory to the rest of society as the practicality crisis. The practicality crisis harms the field in its ability to attract the next generation of scholars and maintain viability at the national level. We describe practical theory and illustrate its use in the field of self-regulation. Psychological theory is historically and scientifically well positioned to become useful should scholars in the field decide to value practicality. We offer a set of incentives to encourage the return of social psychology to the Lewinian vision of a useful science that speaks to pressing social issues.

So Useful as a Good Theory? The Practicality Crisis in (Social) Psychological Theory:

Randomized experiments have enormous potential to improve human welfare in many domains, including healthcare, education, finance, and public policy. However, such "A/B tests" are often criticized on ethical grounds even as similar, untested interventions are implemented without objection. We find robust evidence across 16 studies of 5,873 participants from three diverse populations spanning nine domains-from healthcare to autonomous vehicle design to poverty reduction-that people frequently rate A/B tests designed to establish the comparative effectiveness of two policies or treatments as inappropriate even when universally implementing either A or B, untested, is seen as appropriate. This "A/B effect" is as strong among those with higher educational attainment and science literacy and among relevant professionals. It persists even when there is no reason to prefer A to B and even when recipients are treated unequally and randomly in all conditions (A, B, and A/B). Several remaining explanations for the effect-a belief that consent is required to impose a policy on half of a population but not on the entire population; an aversion to controlled but not to uncontrolled experiments; and a proxy form of the illusion of knowledge (according to which randomized evaluations are unnecessary because experts already do or should know "what works")-appear to contribute to the effect, but none dominates or fully accounts for it. We conclude that rigorously evaluating policies or treatments via pragmatic randomized trials may provoke greater objection than simply implementing those same policies or treatments untested.

https://www.pnas.org/content/pnas/116/22/10723.full.pdf

Objecting to experiments that compare two unobjectionable policies or treatments

In response to our article (1), Mislavsky et al. (2) claim that “experiment aversion” does not exist because they found no evidence of it in their own research on low-stakes corporate experiments (3) and because our studies used between- rather than within-subjects designs.

First, as we noted, we do not expect (and did not ourselves find) an A/B effect in every scenario, and we called for research on how the effect might vary across contexts.

Second, we deliberately used a between-subjects design to maximize external validity: Universal implementation of policies usually occurs without mention of foregone alternatives, whereas A/B tests inherently acknowledge those alternatives. The belief that experiments deprive people of potentially beneficial interventions, but universally implemented policies do not, is not a “confound” to be avoided (2) but, rather, a key … 

[↵][1]1To whom correspondence may be addressed. Email: michellenmeyer{at}gmail.com.

 [1]: #xref-corresp-1-1

Stephen Anderson

Papers

Objecting to experiments that compare two unobjectionable policies or treatments

Reply to Mislavsky et al.: Sometimes people really are averse to experiments