Google Earth Engine

A nonrelational, distributed computing, data warehouse, and analytics environment (Energy-CRADLE) was developed for the analysis of field and laboratory data from multiple heterogeneous photovoltaic (PV) test sites. This data informatics and analytics infrastructure was designed to process diverse formats of PV performance data and climatic telemetry time-series data collected from a PV outdoor test network, i.e., the Solar Durability and Lifetime Extension global SunFarm network, as well as point-in-time laboratory spectral and image measurements of PV material samples. Using Hadoop/HBase for the distributed data warehouse, Energy-CRADLE does not have a predefined data table schema, which enables ingestion of data in diverse and changing formats. For easy data ingestion and data retrieval, Energy-CRADLE utilizes Hadoop streaming to enable Python MapReduce and provides a graphical user interface, i.e., py-CRADLE. By developing the Hadoop distributed computing platform and the HBase NoSQL database schema for solar energy, Energy-CRADLE exemplifies an integrated, scalable, secure, and user-friendly data informatics and analytics system for PV researchers. An example of Energy-CRADLE enabled scalable, data-driven, analytics is presented, where machine learning is used for anomaly detection across 2.2 million real-world current-voltage (I–V) curves of PV modules in three distinct Koppen–Geiger climatic zones.

A Nonrelational Data Warehouse for the Analysis of Field and Laboratory Data From Multiple Heterogeneous Photovoltaic Test Sites

Big geospatial raster data pose a grand challenge to data management technologies for effective big data query and processing. To address these challenges, various big data container solutions have been developed or enhanced to facilitate data storage, retrieval, and analysis. Data containers were also developed or enhanced to handle geospatial data. For example, Rasdaman was developed to handle raster data and GeoSpark/SpatialHadoop were enhanced from Spark/Hadoop to handle vector data. However, there are few studies to systematically compare and evaluate the features and performances of these popular data containers. This paper provides a comprehensive evaluation of six popular data containers (i.e., Rasdaman, SciDB, Spark, ClimateSpark, Hive, and MongoDB) for handling multi-dimensional, array-based geospatial raster datasets. Their architectures, technologies, capabilities, and performance are compared and evaluated from two perspectives: (a) system design and architecture (distributed architecture, logical data model, physical data model, and data operations); and (b) practical use experience and performance (data preprocessing, data uploading, query speed, and resource consumption). Four major conclusions are offered: (1) no data containers, except ClimateSpark, have good support for the HDF data format used in this paper, requiring time- and resource-consuming data preprocessing to load data; (2) SciDB, Rasdaman, and MongoDB handle small/mediate volumes of data query well, whereas Spark and ClimateSpark can handle large volumes of data with stable resource consumption; (3) SciDB and Rasdaman provide mature array-based data operation and analytical functions, while the others lack these functions for users; and (4) SciDB, Spark, and Hive have better support of user defined functions (UDFs) to extend the system capability.

Evaluating the Open Source Data Containers for Handling Big Geospatial Raster Data

Secure NoSQL for the Social Networking and E-Commerce Based Bigdata Applications Deployed in Cloud

The proliferation of Big Data systems and namely NoSQL databases has resulted in a tremendous heterogeneity in its offerings. It has become increasingly difficult to compare and select the most optimal NoSQL storage technology. Current benchmark efforts, such as the Yahoo! Cloud Serving Benchmark (YCSB), evaluate simple read and write operations on a primary key. However, while YCSB has become the de-facto benchmark solution for practitioners and NoSQL vendors, it is lacking in capabilities to extensively evaluate specific NoSQL solutions.In this paper, we present a systematic survey of current NoSQL benchmarks, in which we identify a clear gap in benchmarking more advanced workloads (e.g. nested document search) for features specific to NoSQL database families (e.g. document stores). Secondly, based on our survey, we discuss the strengths and weaknesses of the different benchmark design approaches, and argue in favor of a benchmark suite that targets specific families of NoSQL databases yet still allows overall comparison of databases in terms of their commonalities.

On the State of NoSQL Benchmarks

Analyses in climate research typically operate on large datasets stored in file hierarchies. However, e.g. Indexing, meta-data search and replication is challenging with this approach. A new approach is to store and index datasets in large, distributed databases. In order to demonstrate the performance improvement, we utilized the so-called general matching problem between measurements of two satellites that differ in orbits and measurement cycles. For comparison purpose the measurements are matched within a specified maximum spatial and time offset constraint. We describe the steps from a single-threaded approach using a SQL database to a multi-threaded approach using the NoSQL database MongoDB. The performance as well as limitations on CPU and I/O are evaluated and discussed for each approach. The performance has been improved up to a factor of 46 using 11 threads on a 12-core system.

On the Application and Performance of MongoDB for Climate Satellite Data

The evaluation and benchmarking of NoSQL databases is challenging due to the variety of query languages and also due to the denormalized scheme that allows to include non-scalar values and repeatable structures such as nested documents. In the rather young age of NoSQL databases, there are not many studies on a conventional solution to enable a fair comparison of their performance with each other and with relational databases. In this paper, we present the modeling and evaluation of an extensible open-source solution called NoWog. Its purpose is to generate large synthetic database workloads and also to mimic real-world database applications allowing application-specific benchmarking. We introduce a generic language for the description of workload characteristics that is independent of a particular database query language or the underlying storage model and therefore applicable to various database management systems. Thereby, the NoWog language covers different distributions of read and write operations for various data types such as numbers, texts, arrays and nested documents. In our evaluation, we demonstrate NoWog's ability to mimic a real workload from an application using environmental satellite data stored in a MongoDB and also its scalability to produce big workloads.

NoWog: A Workload Generator for Database Performance Benchmarking

Due to wide usage of databases and ever increasing size of their datasets and workloads, tuning the physical database design to improve performance with regard to specific workloads is crucial. An important aspect of physical database design is selecting fitting indexes to improve run-time of a workload. The proper set of indexes should be chosen by considering ratio of read to write of workload operations, selectivity of attributes and resource limitations. Since databases have their own cost-functions implemented in their query optimizer, a reasonable way of choosing indexes is to present a limited number of candidate indexes to the optimizer and recommend its chosen ones. However, this selection process should not put too much load on the in-production system. Therefore, the focus of this thesis is on utilizing scalable algorithms to propose minimum and yet required number of the candidate indexes for each workload to the query optimizer. We recommend indexes only for longest and most frequent queries in the workload. The indexes are prioritized by their coverage for more queries, existence in the system and selectivity of their attributes in the dataset. We also propose a method to avoid interference of the index recommendation process with other applications.

On a self-tuning index recommendation approach for databases

We present a distributed system for storage, processing, three-dimensional visualisation and basic analysis of data from Earth-observing satellites. The database and the server have been designed for high performance and scalability, whereas the client is highly portable thanks to having been designed as a HTML5- and WebGL-based Web application. The system is based on the so-called MEAN stack, a modern replacement for LAMP which has steadily been gaining traction among high-performance Web applications. We demonstrate the performance of the system from the perspective of an user operating the client.

A distributed system for storing and processing data from earth-observing satellites: system design and performance evaluation of the visualisation tool

The remote sensing of atmospheric trace gases investigates dynamic, microphysical and chemical processes in the Earth’s atmosphere, with the goal to understand, quantify and predict its natural variability and long-term changes. Accurate measurements of atmospheric trace gases from various observational platforms (ground-based stations, air craft, balloons, satellites) provide the data that are required for the modelling of atmospheric processes. The instrument GLORIA (Gimballed Limb Observer for Radiance Imaging of the Atmosphere), developed by KIT/IMK and FZ Julich, an Infrared Spectrometer, which measures atmospheric emissions, was engaged in several measurement campaigns on board of HALO (High Altitude and Long Range Research Aircraft) and provided a large amount of data, which has to be managed efficiently for processing and visualisation. This paper describes the system background and the use of MongoDB for the provision of measured and processed mass data.

/pdf/management-of-meteorological-mass-data-with-mongodb-4b7s9ln28a.pdf

Parinaz Ameri

Papers

On the Application and Performance of MongoDB for Climate Satellite Data

NoWog: A Workload Generator for Database Performance Benchmarking

On a self-tuning index recommendation approach for databases

A distributed system for storing and processing data from earth-observing satellites: system design and performance evaluation of the visualisation tool

Management of Meteorological Mass Data with MongoDB