scispace - formally typeset
Search or ask a question
Journal Article•DOI•

Critical questions for big data

danah boyd1, Kate Crawford1•
Microsoft1
25 May 2012-Information, Communication & Society (Informa UK Limited)-Vol. 15, Iss: 5, pp 662-679
TL;DR: The era of Big Data has begun as discussed by the authors, where diverse groups argue about the potential benefits and costs of analyzing genetic sequences, social media interactions, health records, phone logs, government records, and other digital traces left by people.
Abstract: The era of Big Data has begun. Computer scientists, physicists, economists, mathematicians, political scientists, bio-informaticists, sociologists, and other scholars are clamoring for access to the massive quantities of information produced by and about people, things, and their interactions. Diverse groups argue about the potential benefits and costs of analyzing genetic sequences, social media interactions, health records, phone logs, government records, and other digital traces left by people. Significant questions emerge. Will large-scale search data help us create better tools, services, and public goods? Or will it usher in a new wave of privacy incursions and invasive marketing? Will data analytics help us understand online communities and political movements? Or will it be used to track protesters and suppress speech? Will it transform how we study human communication and culture, or narrow the palette of research options and alter what ‘research’ means? Given the rise of Big Data as a socio-tech...

Summary (3 min read)

Jump to: [AGRADECIMIENTOS] – [INTRODUCCIÓN] – [Tabla 2. Comparación de propiedades Base y Acid BASE ACID] – [Fuente: Autores] – [Tabla 7. Uso de SQL y NoSQL] – [Cloud] – [Está basado en dos estructuras:] and [1.7.20 Multi-master replication.]

AGRADECIMIENTOS

  • Agradecemos en primer lugar a Dios por permitirnos llegar a esta etapa de nuestras vidas, reflejando a través de este trabajo todas las enseñanzas recibidas durante el transcurso de nuestra carrera.
  • En segundo lugar agradecemos la compañía de nuestras familias, por apoyarnos en cada momento, brindando su protección y ayuda para tomar las mejores decisiones.
  • Por último, agradecemos a nuestros amigos y compañeros de universidad y a todos los maestros que tuvimos durante la carrera; también agradecemos a la profesora Claudia Milena Rodríguez Álvarez quien nos guio en gran parte del proyecto y al profesor Yasser de Jesús Muriel Perea, quien en esta última fase acepto asesorarnos para la finalización del proyecto.
  • Anexo A. Guía para la creación del ambiente Big Data 57 Anexo B. Guía para la implementación del caso de estudio 79.

INTRODUCCIÓN

  • Con el constante crecimiento de información en cada uno de los aspectos más relevantes mundialmente como sociedad, comercio y ciencia, se vuelve necesario un cambio con respecto al manejo de la información, que hasta hace poco tiempo se venía implementando a partir de bases de datos relacionales.
  • "Actualmente se pueden encontrar tecnologías como Hadoop, MapReduce y bases de datos NoSQL, que se pueden implementar en la creación de un ambiente Big Data" 2 .
  • De esta manera, el proyecto busco la implementación de una arquitectura para crear un ambiente Big Data en la Universidad Católica de Colombia, teniendo en cuenta aspectos importantes como el software y el hardware que se debía utilizar para realizar dicha labor, de igual manera todos los procedimientos que implicaba empezar a utilizar bases de datos no relacionales.
  • Todo esto se ve reflejado a través de dos guías, donde se podrá encontrar de una manera detallada los pasos para la implementación de un ambiente Big Data, sus elementos y procesos para ingresar y consultar información a través de un ejemplo específico, con el propósito de diseñar estrategias y buscar patrones que permitan una buena gestión de la información.

Tabla 2. Comparación de propiedades Base y Acid BASE ACID

  • Esto quiere decir básicamente disponible, aquí se utiliza la replicación para reducir la probabilidad de que se presente la indisponibilidad de los datos o la fragmentación de la información a través de los distintos servidores de almacenamiento, also known as Basically Available.
  • Toda la secuencia de acciones debe ser completada o abortada, es decir que la operación no puede ser parcialmente completada, debe haber una operación de commit cuando se completa o una de rollback en el momento en que la transacción no pueda ser completada, also known as Atomicidad.
  • Esto quiere decir que la transacción toma los recursos de un estado valido para llevar a la base de datos a otro estado valido, also known as Consistencia.
  • Aunque las aplicaciones deben asumir la consistencia como un requerimiento de alta prioridad, "los sistemas NoSQL garantizan que en algún punto del futuro, los datos asumen un estado de consistencia, haciendo comparación con los sistemas relacionales que obligan que una transacción sea completada" 33 , los sistemas NoSQL brindan consistencia a lo largo del tiempo, also known as Eventualmente Consistente.
  • Todas las transacciones realizadas deben ser permanentes y ser tolerantes a los fallos del sistema, also known as Durabilidad.

Fuente: Autores

  • A continuación se realiza un cuadro comparativo sobre algunos motores de bases de datos NoSQL, que se pueden utilizar para la construcción de un ambiente Big Data, teniendo en cuenta su taxonomía.
  • Hive permite a los usuarios leer los datos en formatos arbitrarios, usando SerDes y entrada / Formatos de salida 45 .
  • 6.14 Transformación de una base de datos relacional a NoSQL.
  • Según Couchbase 49 con la publicidad que ha tenido el concepto de una base de datos no relacional que ha sido abrumadora, se tiene un concepto que se encuentra por debajo de las demandas que a menudo son exageradas, debido a que como la mayoría de cosas los beneficios tienen un costo.
  • Como se muestra en la figura anterior, según Couchbase 52 cada registro en una base de datos relacional conforma un esquema según un número establecido de 51 Ibid.

Tabla 7. Uso de SQL y NoSQL

  • Ámbitos de uso para SQL Ámbitos de uso para NOSQL Educativo: para aportar a los estudiantes conocimientos acerca de la estructuración de información.
  • Tabla 7. Desarrollo Web: para mantener una misma jerarquía de los datos que llegan de la gran autopista, pero siempre y cuando la capacidad de concurrencia, almacenamiento y mantenimiento no sean de considerable dificultad y la información siempre sea consistente.
  • En estos momentos, las empresas están lidiando con un problema grande conocido como Bring Your Own Device en realidad no es un problema, es un fenómeno social, por lo que la información que se recolecte siempre será diferente por más que uno desee estructurarla y mantenerla estática.
  • Para inteligencia de negocios, análisis de negocios, bodegas de datos, minería de datos, minería de texto son temas que requieren el uso de SQL para facilitar el consumo de la información y la identificación de patrones en los datos, also known as Rama de negocios.
  • Para el software a la medida y el software empresarial, ambos de escritorio, poseen la característica de mantener información con una estructura consistente y SQL es ideal para esta tarea, also known as Empresarial.

Cloud

  • Termino XaaS (Everything as a service) que indica "cualquier cosa como servicio (sic)" y todos los temas relacionados en la nube, con NoSQL pueden adaptarse casi a cualquier necesidad del cliente, que evidentemente son heterogéneos, also known as (XaaS).
  • CouchDB es una base de datos que abarca completamente la red, utiliza documentos en JSON para guardar los datos, permite acceder a los datos desde un navegador web a través del protocolo http, permite realizar operaciones utilizando JavaScript 64 .
  • Las bases de datos NoSQL "representan una evolución en la arquitectura de aplicación del negocio, están diseñadas para proveer el almacenamiento de datos confiables, escalables y disponibles a través de un conjunto de sistemas configurables que funcionan como nodos de almacenamiento" 66 . 1.7.4 Big Data.
  • Fue creado a partir del Google File System (GFS).
  • Es un formato ligero de intercambio de datos, está basado en un subconjunto del lenguaje de programación JavaScript.

Está basado en dos estructuras:

  • En varios lenguajes esto se realiza como un registro, estructura, o arreglo asociado a objetos.
  • Como la secuencia del nombre MapReduce indica el "reduce Job" siempre se ejecutará después de que el "map Job" haya sido ejecutado 79 .
  • "Es un modelo para una comunicación de protocolo en el que un dispositivo o proceso (conocido como el maestro) controla uno o más de otros dispositivos o procesos (conocida como esclavos).
  • El modelo de datos entidadrelación (E-R) es útil para hacer corresponder los significados e interacciones de las empresas del mundo real con un esquema conceptual" 81 .

1.7.20 Multi-master replication.

  • En una configuración multi-master, los datos se actualizan en varios patrones.
  • Cada maestro mantiene un registro de cambios, y los cambios realizados en cada master se replican en los demás servidores.
  • Cuando se envían los cambios entre los dos servidores, cualquier cambio en conflicto debe ser resuelto 83 .
  • "Medios para dividir los datos de tal manera que los datos solicitados y actualizados estén en el mismo nodo y que el volumen de carga y almacenamiento se distribuyan entre los servidores.
  • Los fragmentos de datos también pueden ser replicados por razones de fiabilidad y de equilibrio de carga y pueden esperarse a escribir en sólo una réplica dedicado o a todas las réplicas que mantienen una partición de los datos" 84 .

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

DISEÑO Y DESARROLLO DE UNA GUÍA PARA LA IMPLEMENTACIÓN DE UN
AMBIENTE BIG DATA EN LA UNIVERSIDAD CATÓLICA DE COLOMBIA
FABIÁN ANDRÉS GUERRERO LÓPEZ
JORGE EDUARDO RODRÍGUEZ PINILLA
UNIVERSIDAD CATÓLICA DE COLOMBIA
FACULTAD DE INGENIERÍA
PROGRAMA DE INGENIERÍA DE SISTEMAS
MODALIDAD TRABAJO DE INVESTIGACIÓN
BOGOTÁ
2013
DISEÑO Y DESARROLLO DE UNA GUÍA PARA LA IMPLEMENTACIÓN DE UN
AMBIENTE BIG DATA EN LA UNIVERSIDAD CATÓLICA DE COLOMBIA
FABIÁN ANDRÉS GUERRERO LÓPEZ
JORGE EDUARDO RODRÍGUEZ PINILLA
Trabajo de Grado para optar al título de
Ingeniero de Sistemas
Director
YASSER DE JESÚS MURIEL PEREA
Ingeniero de Sistemas
UNIVERSIDAD CATÓLICA DE COLOMBIA
FACULTAD DE INGENIERÍA
PROGRAMA DE INGENIERÍA DE SISTEMAS
MODALIDAD TRABAJO DE INVESTIGACIÓN
BOGOTÁ
2013
3
4
Nota de aceptación
Aprobado por el comité de grado
en cumplimiento de los requisitos
Exigidos por la Facultad de
Ingeniería y la Universidad Católica
de Colombia para optar al título de
Ingenieros de Sistemas.
________________________________
Director
____________________________
Revisor Metodológico
Bogotá, 27 de Noviembre, 2013
5
AGRADECIMIENTOS
Agradecemos en primer lugar a Dios por permitirnos llegar a esta etapa de
nuestras vidas, reflejando a través de este trabajo todas las enseñanzas recibidas
durante el transcurso de nuestra carrera.
En segundo lugar agradecemos la compañía de nuestras familias, por apoyarnos
en cada momento, brindando su protección y ayuda para tomar las mejores
decisiones.
Por último, agradecemos a nuestros amigos y compañeros de universidad y a
todos los maestros que tuvimos durante la carrera; también agradecemos a la
profesora Claudia Milena Rodríguez Álvarez quien nos guio en gran parte del
proyecto y al profesor Yasser de Jesús Muriel Perea, quien en esta última fase
acepto asesorarnos para la finalización del proyecto.
Citations
More filters
Journal Article•DOI•
The Parable of Google Flu: Traps in Big Data Analysis

[...]

David Lazer1, Ryan Kennedy2, Ryan Kennedy1, Ryan Kennedy3, Gary King2, Alessandro Vespignani1, Alessandro Vespignani2 •
Northeastern University1, Harvard University2, University of Houston3
14 Mar 2014-Science
TL;DR: Large errors in flu prediction were largely avoidable, which offers lessons for the use of big data.
Abstract: In February 2013, Google Flu Trends (GFT) made headlines but not for a reason that Google executives or the creators of the flu tracking system would have hoped. Nature reported that GFT was predicting more than double the proportion of doctor visits for influenza-like illness (ILI) than the Centers for Disease Control and Prevention (CDC), which bases its estimates on surveillance reports from laboratories across the United States ( 1 , 2 ). This happened despite the fact that GFT was built to predict CDC reports. Given that GFT is often held up as an exemplary use of big data ( 3 , 4 ), what lessons can we draw from this error?

2,062 citations

Journal Article•DOI•
The real-time city? Big data and smart urbanism

[...]

Rob Kitchin1•
Maynooth University1
01 Feb 2014-GeoJournal
TL;DR: In this article, the authors focus on the implications of big data and smart urbanism, examining five emerging concerns: the politics of big urban data, technocratic governance and city development, corporatisation of city governance and technological lock-ins, buggy, brittle and hackable cities, and the panoptic city.
Abstract: ‘Smart cities’ is a term that has gained traction in academia, business and government to describe cities that, on the one hand, are increasingly composed of and monitored by pervasive and ubiquitous computing and, on the other, whose economy and governance is being driven by innovation, creativity and entrepreneurship, enacted by smart people. This paper focuses on the former and, drawing on a number of examples, details how cities are being instrumented with digital devices and infrastructure that produce ‘big data’. Such data, smart city advocates argue enables real-time analysis of city life, new modes of urban governance, and provides the raw material for envisioning and enacting more efficient, sustainable, competitive, productive, open and transparent cities. The final section of the paper provides a critical reflection on the implications of big data and smart urbanism, examining five emerging concerns: the politics of big urban data, technocratic governance and city development, corporatisation of city governance and technological lock-ins, buggy, brittle and hackable cities, and the panoptic city.

1,475 citations

Journal Article•DOI•
Big Data, new epistemologies and paradigm shifts:

[...]

Rob Kitchin1•
Maynooth University1
01 Apr 2014-Big Data & Society
TL;DR: The authors examines how the availability of Big Data, coupled with new data analytics, challenges established epistemologies across the sciences, social sciences and humanities, and assesses the extent to which they are engendering paradigm shifts across multiple disciplines.
Abstract: This article examines how the availability of Big Data, coupled with new data analytics, challenges established epistemologies across the sciences, social sciences and humanities, and assesses the extent to which they are engendering paradigm shifts across multiple disciplines. In particular, it critically explores new forms of empiricism that declare ‘the end of theory’, the creation of data-driven rather than knowledge-driven science, and the development of digital humanities and computational social sciences that propose radically different ways to make sense of culture, history, economy and society. It is argued that: (1) Big Data and new data analytics are disruptive innovations which are reconfiguring in many instances how research is conducted; and (2) there is an urgent need for wider critical reflection within the academy on the epistemological implications of the unfolding data revolution, a task that has barely begun to be tackled despite the rapid changes in research practices presently taking place. After critically reviewing emerging epistemological positions, it is contended that a potentially fruitful approach would be the development of a situated, reflexive and contextually nuanced epistemology.

1,463 citations

Cites background from "Critical questions for big data"

  • ...(see boyd and Crawford, 2012; Dodge and Kitchin, 2005; Laney, 2001; Marz and Warren, 2012; Mayer-Schonberger and Cukier, 2013; Zikopoulos et al., 2012)....

    [...]

  • ...(boyd and Crawford, 2012) As with many rapidly emerging concepts, Big Data has been variously defined and operationalized, ranging from trite proclamations that Big Data consists of datasets too large to fit in an Excel spreadsheet or be stored on a single machine (Strom, 2012) to…...

    [...]

  • ...…trite proclamations that Big Data consists of datasets too large to fit in an Excel spreadsheet or be stored on a single machine (Strom, 2012) to more sophisticated ontological assessments that tease out its inherent characteristics (boyd and Crawford, 2012; Mayer-Schonberger and Cukier, 2013)....

    [...]

Journal Article•DOI•
Critical analysis of Big Data challenges and analytical methods

[...]

Uthayasankar Sivarajah1, Muhammad Kamal1, Zahir Irani1, Vishanth Weerakkody1•
Brunel University London1
01 Jan 2017-Journal of Business Research
TL;DR: In this article, the authors present a state-of-the-art review that presents a holistic view of the BD challenges and BDA methods theorized/proposed/employed by organizations to help others understand this landscape with the objective of making robust investment decisions.

1,267 citations

Cites background from "Critical questions for big data"

  • ...…challenges have paid attention to the difficulties of understanding the notion of BD (Hargittai, 2015), decision-making of what data are generated and collected (Crawford, 2013), issues of privacy (Lazer et al., 2009) and ethical considerations relevant to mining such data (Boyd & Crawford, 2012)....

    [...]

The Relevance of Algorithms

[...]

Tarleton Gillespie, Pablo J. Boczkowski
01 Jan 2013
TL;DR: This chapter contains section titled: Patterns of Inclusion, Cycles of Anticipation, The Evaluation of Relevance, The Promise of Algorithmic Objectivity, Entanglement with Practice, and The Production of Calculated Publics.
Abstract: This chapter contains section titled: Patterns of Inclusion, Cycles of Anticipation, The Evaluation of Relevance, The Promise of Algorithmic Objectivity, Entanglement with Practice, The Production of Calculated Publics, Conclusion, Acknowledgments, Notes

1,133 citations

References
More filters
Journal Article•DOI•
The Strength of Weak Ties

[...]

Mark Granovetter1•
Johns Hopkins University1
01 May 1973-American Journal of Sociology
TL;DR: In this paper, it is argued that the degree of overlap of two individuals' friendship networks varies directly with the strength of their tie to one another, and the impact of this principle on diffusion of influence and information, mobility opportunity, and community organization is explored.
Abstract: Analysis of social networks is suggested as a tool for linking micro and macro levels of sociological theory. The procedure is illustrated by elaboration of the macro implications of one aspect of small-scale interaction: the strength of dyadic ties. It is argued that the degree of overlap of two individuals' friendship networks varies directly with the strength of their tie to one another. The impact of this principle on diffusion of influence and information, mobility opportunity, and community organization is explored. Stress is laid on the cohesive power of weak ties. Most network models deal, implicitly, with strong ties, thus confining their applicability to small, well-defined groups. Emphasis on weak ties lends itself to discussion of relations between groups and to analysis of segments of social structure not easily defined in terms of primary groups.

37,560 citations

Book•
Social behavior; its elementary forms

[...]

George C. Homans1•
Harvard University1
01 Jan 1961

5,819 citations

Book•
Writing Culture: The Poetics and Politics of Ethnography

[...]

James Clifford, George E. Marcus
01 Jan 1986
TL;DR: The authors explore the ways in which writing culture has changed the face of ethnography over the last 25 years. But they do not discuss the role of writing culture in the development of ethnographies.
Abstract: This seminal collection of essays critiquing ethnography as literature is augmented with a new foreword by Kim Fortun, exploring the ways in which Writing Culture has changed the face of ethnography over the last 25 years.

5,353 citations

"Critical questions for big data" refers background in this paper

  • ...Social scientists have a long history of asking critical questions about the collection of data and trying to account for any biases in their data (Cain & Finch 1981 ; Clifford & Marcus 1986 )....

    [...]

  • ...Social scientists have a long history of asking critical questions about the collection of data and trying to account for any biases in their data (Cain & Finch 1981; Clifford & Marcus 1986)....

    [...]

Book•
The Rules of Sociological Method

[...]

Emile Durkheim, Steven Lukes
01 Jan 1895
TL;DR: The Rules of the Sociological Method as discussed by the authors is one of the most important contributions to the field of sociology, still debated among scholars today, and has been a focal point of sociology since its original publication.
Abstract: First published in 1895: Emile Durkheim's masterful work on the nature and scope of sociology--now with a new introduction and improved translation by leading scholar Steven Lukes.The Rules of the Sociological Method is among the most important contributions to the field of sociology, still debated among scholars today. Through letters, arguments, and commentaries on significant debates, Durkheim confronted critics, clarified his own position, and defended the objective scientific method he applied to his study of humans. This updated edition offers an introduction and extra notes as well as a new translation to improve the clarity and accessibility of this essential work. In the introduction, Steven Lukes, author of the definitive biography Emile Durkheim: His Life and Work, spells out Durkheim's intentions, shows the limits of Durkheim's view of sociology, and presents its political background and significance. Making use of the various texts in this volume and Durkheim's later work, Lukes discusses how Durkheim's methodology was modified or disregarded in practice--and how it is still relevant today. With substantial notes on context, this user-friendly edition will greatly ease the task of students and scholars working with Durkheim's method--a view that has been a focal point of sociology since its original publication. The Rules of the Sociological Method will engage a new generation of readers with Durkheim's rich contribution to the field."

3,876 citations

Book•
Code and Other Laws of Cyberspace

[...]

Lawrence Lessig
01 Jan 1999
TL;DR: Harvard Professor Lawrence Lessig shows how code can make a domain, site, or network free or restrictive; how technological architectures influence people's behavior and the values they adopt; and how changes in code can have damaging consequences for individual freedoms.
Abstract: From the Publisher: Should cyberspace be regulated? How can it be done? It's a cherished belief of techies and net denizens everywhere that cyberspace is fundamentally impossible to regulate. Harvard Professor Lawrence Lessig warns that, if we're not careful we'll wake up one day to discover that the character of cyberspace has changed from under us. Cyberspace will no longer be a world of relative freedom; instead it will be a world of perfect control where our identities, actions, and desires are monitored, tracked, and analyzed for the latest market research report. Commercial forces will dictate the change, and architecture—the very structure of cyberspace itself—will dictate the form our interactions can and cannot take. Code And Other Laws of Cyberspace is an exciting examination of how the core values of cyberspace as we know it—intellectual property, free speech, and privacy-—are being threatened and what we can do to protect them. Lessig shows how code—the architecture and law of cyberspace—can make a domain, site, or network free or restrictive; how technological architectures influence people's behavior and the values they adopt; and how changes in code can have damaging consequences for individual freedoms. Code is not just for lawyers and policymakers; it is a must-read for everyone concerned with survival of democratic values in the Information Age.

2,706 citations

"Critical questions for big data" refers background in this paper

  • ...Lessig (1999) argues that social systems are regulated by four forces: market, law, social norms, and architecture – or, in the case of technology, code....

    [...]

Related Papers (5)
Big Data: A Revolution That Will Transform How We Live, Work, and Think

[...]

05 Mar 2013
Viktor Mayer-Schnberger, Kenneth Cukier
Business intelligence and analytics: from big data to big impact

[...]

01 Dec 2012-Management Information Systems Quarterly
Hsinchun Chen, Roger H. L. Chiang, Veda C. Storey
Big data: the management revolution.

[...]

01 Oct 2012-Harvard Business Review
Andrew McAfee, Erik Brynjolfsson
Computational Social Science

[...]

06 Feb 2009-Science
David Lazer, Alex Pentland, Lada A. Adamic, Sinan Aral, Sinan Aral, Albert-László Barabási, Devon Brewer, Nicholas A. Christakis, Noshir Contractor, James H. Fowler, Myron P. Gutmann, Tony Jebara, Gary King, Michael W. Macy, Deb Roy, Marshall Van Alstyne, Marshall Van Alstyne 
The Parable of Google Flu: Traps in Big Data Analysis

[...]

14 Mar 2014-Science
David Lazer, Ryan Kennedy, Ryan Kennedy, Ryan Kennedy, Gary King, Alessandro Vespignani, Alessandro Vespignani 
Frequently Asked Questions (1)
Q1. What is the importance of the two models?

Para aclarar las diferencias que pueden existir dentro de las bases de datos relaciones y las NoSQL, se debe comprender dos modelos importantes: Base y Acid, los cuales se utilizan respectivamente para el manejo de las transacciones.