From the edge to the cloud: A continuous delivery and preparation model for processing big IoT data

An efficient pattern-based approach for workflow supporting large-scale science: The DagOnStar experience

The impact of machine learning algorithms on everyday life is overwhelming until the novel concept of datacracy as a new social paradigm. In the field of computational environmental science and, in particular, of applications of large data science proof of concept on the natural resources management this kind of approaches could make the difference between species surviving to potential extinction and compromised ecological niches. In this scenario, the use of high throughput workflow engines, enabling the management of complex data flows in production is rock solid, as demonstrated by the rise of recent tools as Parsl and DagOnStar. Nevertheless, the availability of dedicated computational resources, although mitigated by the use of cloud computing technologies, could be a remarkable limitation. In this paper, we present a novel and improved version of DagOnStar, enabling the execution of lightweight but recurring computational tasks on the microservice architecture. We present our preliminary results motivating our choices supported by some evaluations and a real-world use case.

A Microservice-Based Building Block Approach for Scientific Workflow Engines: Processing Large Data Volumes with DagOnStar

In this paper, we present the design, implementation, and evaluation of an efficient data preparation and retrieval approach for cloud storage. The approach includes a deduplication subsystem that indexes the hash of each content to identify duplicated data. As a consequence, avoiding duplicated content reduces reprocessing time during uploads and other costs related to outsource data management tasks. Our proposed data preparation scheme enables organizations to add properties such as security, reliability, and cost-efficiency to their contents before sending them to the cloud. It also creates recovery schemes for organizations to share preprocessed contents with partners and end-users. The approach also includes an engine that encapsulates preprocessing applications into virtual containers (VCs) to create parallel patterns that improve the efficiency of data preparation retrieval process. In a study case, real repositories of satellite images, and organizational files were prepared to be migrated to the cloud by using processes such as compression, encryption, encoding for fault tolerance, and access control. The experimental evaluation revealed the feasibility of using a data preparation approach for organizations to mitigate risks that still could arise in the cloud. It also revealed the efficiency of the deduplication process to reduce data preparation tasks and the efficacy of parallel patterns to improve the end-user service experience.

A Data Preparation Approach for Cloud Storage Based on Containerized Parallel Patterns

Brazilian tropical ecosystems in the state of Mato Grosso have experienced significant land use and cover changes during the past few decades due to deforestation and wildfire. These changes can directly affect the mass and energy exchange near the surface and, consequently, evapotranspiration (ET). Characterization of the seasonal patterns of ET can help in understanding how these tropical ecosystems function with a changing climate. The goal of this study was to characterize temporal (seasonal-to-decadal) and spatial patterns in ET over Mato Grosso using remotely sensed products. Ecosystems over areas with limited to no flux towers can be performed using remote sensing products such as NASA’s MOD16A2 ET (MOD16 ET). As the accuracy of this product in tropical ecosystems is unknown, a secondary objective of this study was to evaluate the ability of the MOD16 ET (ETMODIS) to appropriately represent the spatial and seasonal ET patterns in Mato Grosso, Brazil. Actual ET was measured (ETMeasured) using eight flux towers, three in the Amazon, three in the Cerrado, and two in the Pantanal of Mato Grosso. In general, the ETMODIS of all sites had no significant difference from ETMeasured during all analyzed periods, and ETMODIS had a significant moderate to strong correlation with the ETMeasured. The spatial variation of ET had some similarity to the climatology of Mato Grosso, with higher ET in the mid to southern parts of Mato Grosso (Cerrado and Pantanal) during the wet period compared to the dry period. The ET in the Amazon had three seasonal patterns, a higher and lower ET in the wet season compared to the dry season, and minimal to insignificant variation in ET during the wet and dry seasons. The wet season ET in Amazon decreased from the first and second decades, but the ET during the wet and dry season increased in Cerrado and Pantanal in the same period. This study highlights the importance of deepening the study of ET in the state of Mato Grosso due to the land cover and climate change.

/pdf/evapotranspiration-seasonality-over-tropical-ecosystems-in-2bp3z55f.pdf

Evapotranspiration Seasonality over Tropical Ecosystems in Mato Grosso, Brazil

Remote sensing-based evapotranspiration (ET) products have been evaluated primarily using data from northern middle latitudes; therefore, little is known about their performance at low latitudes. To address this bias, an evaluation dataset was compiled using eddy covariance data from 40 sites between latitudes 30° S and 30° N. The flux data were obtained from the emerging network in Mexico (MexFlux) and from openly available databases of FLUXNET, AsiaFlux, and OzFlux. This unique reference dataset was then used to evaluate remote sensing-based ET products in environments that have been underrepresented in earlier studies. The evaluated products were: MODIS ET (MOD16, both the discontinued collection 5 (C5) and the latest collection (C6)), Global Land Evaporation Amsterdam Model (GLEAM) ET, and Atmosphere-Land Exchange Inverse (ALEXI) ET. Products were compared with unadjusted fluxes (ETorig) and with fluxes corrected for the lack of energy balance closure (ETebc). Three common statistical metrics were used: coefficient of determination (R2), root mean square error (RMSE), and percent bias (PBIAS). The effect of a vegetation mismatch between pixel and site on product evaluation results was investigated by examining the relationship between the statistical metrics and product-specific vegetation match indexes. Evaluation results of this study and those published in the literature were used to examine the performance of the products across latitudes. Differences between the MOD16 collection 5 and 6 datasets were generally smaller than differences with the other products. Performance and ranking of the evaluated products depended on whether ETorig or ETebc was used. When using ETorig, GLEAM generally had the highest R2, smallest PBIAS, and best RMSE values across the studied land cover types and climate zones. Neither MOD16 nor ALEXI performed consistently better than the other. When using ETebc, none of the products stood out in terms of both low bias and strong correlations. The use of ETebc instead of ETorig affected the biases more than the correlations. The product evaluation results showed no significant relationship with the degree of match between the vegetation at the pixel and site scale. The latitudinal comparison showed tendencies of lower R2 (all products) but better PBIAS and normalized RMSE values (MOD16 and GLEAM) for forests at low latitudes than for forests at northern middle latitudes. For non-forest vegetation, the products showed no clear latitudinal differences in performance. 

Evaluation of remote sensing-based evapotranspiration products at low-latitude eddy covariance sites

This paper presents a containerized service for clustering and categorization of weather records in the cloud. This service considers a scheme of microservices and containers for organizations and end-users to manage/process weather records from the acquisition, passing through the prepossessing and processing stages, to the exhibition of results. In this service, a specialized crawler acquires records that are delivered to a microservice of distributed categorization of weather records, which performs clustering of acquired data (the temperature and precipitation) by spatiotemporal parameters. The clusters found are exhibited in a map by a geoportal where statistic microservice also produce results regression graphs on-the-fly. To evaluate the feasibility of this service, a case study based on 33 years of daily records captured by the Mexican weather station network (EMAS-CONAGUA) has been conducted. Lessons learned in this study about the performance of record acquisition, clustering processing, and mapping exhibition are described in this paper. Examples of utilization of this service revealed that end-users can analyze weather parameters in an efficient, flexible and automatic manner.

Susana Alvarado-Barrientos

Papers

Evaluation of remote sensing-based evapotranspiration products at low-latitude eddy covariance sites

A containerized service for clustering and categorization of weather records in the cloud