Data Mining - Concepts and Techniques.

We design flexible schemes to explore the tradeoffs between storage space and access efficiency in reliable data storage systems. Aiming at this goal, two new classes of erasure-resilient codes are introduced -- Basic Pyramid Codes (BPC) and Generalized Pyramid Codes (GPC). Both schemes require slightly more storage space than conventional schemes, but significantly improve the critical performance of read during failures and unavailability.As a by-product, we establish a necessary matching condition to characterize the limit of failure recovery, that is, unless the matching condition is satisfied, a failure case is impossible to recover. In addition, we define a maximally recoverable (MR) property. For all ERC schemes holding the MR property, the matching condition becomes sufficient, that is, all failure cases satisfying the matching condition are indeed recoverable. We show that GPC is the first class of non-MDS schemes holding the MR property.

/pdf/pyramid-codes-flexible-schemes-to-trade-space-for-access-46ge2xdgv6.pdf

Pyramid Codes: Flexible Schemes to Trade Space for Access Efficiency in Reliable Data Storage Systems

An ensemble-based model of PM2.5 concentration across the contiguous United States with high spatiotemporal resolution

https://nottingham-repository.worktribe.com/preview/949995/ChenEtAl18b_authorVersion.pdf

A machine learning method to estimate PM2.5 concentrations across China with remote sensing, meteorological and land use information.

Estimating 1-km-resolution PM2.5 concentrations across China using the space-time random forest approach

To estimate PM25 concentrations, many parametric regression models have been developed, while nonparametric machine learning algorithms are used less often and national-scale models are rare In this paper, we develop a random forest model incorporating aerosol optical depth (AOD) data, meteorological fields, and land use variables to estimate daily 24 h averaged ground-level PM25 concentrations over the conterminous United States in 2011 Random forests are an ensemble learning method that provides predictions with high accuracy and interpretability Our results achieve an overall cross-validation (CV) R2 value of 080 Mean prediction error (MPE) and root mean squared prediction error (RMSPE) for daily predictions are 178 and 283 μg/m3, respectively, indicating a good agreement between CV predictions and observations The prediction accuracy of our model is similar to those reported in previous studies using neural networks or regression models on both national and regional scales In addition, the 

Estimating PM2.5 Concentrations in the Conterminous United States Using the Random Forest Approach.

Low-cost air quality sensors are promising supplements to regulatory monitors for PM2.5 exposure assessment. However, little has been done to incorporate the low-cost sensor measurements in large-scale PM2.5 exposure modeling. We conducted spatially varying calibration and developed a downweighting strategy to optimize the use of low-cost sensor data in PM2.5 estimation. In California, PurpleAir low-cost sensors were paired with air quality system (AQS) regulatory stations, and calibration of the sensors was performed by geographically weighted regression. The calibrated PurpleAir measurements were then given lower weights according to their residual errors and fused with AQS measurements into a random forest model to generate 1 km daily PM2.5 estimates. The calibration reduced PurpleAir's systematic bias to ∼0 μg/m3 and residual errors by 36%. Increased sensor bias was found to be associated with higher temperature and humidity, as well as longer operating time. The weighted prediction model outperformed the AQS-based prediction model with an improved random cross-validation (CV) R2 of 0.86, an improved spatial CV R2 of 0.81, and a lower prediction error. The temporal CV R2 did not improve due to the temporal discontinuity of PurpleAir. The inclusion of PurpleAir data allowed the predictions to better reflect PM2.5 spatial details and hotspots.

Incorporating Low-Cost Sensor Measurements into High-Resolution PM2.5 Modeling at a Large Spatial Scale.

Impacts of snow and cloud covers on satellite-derived PM2.5 levels.

Deduplicating in-line data on primary storage is hampered by the disk bottleneck problem, an issue which results from the need to keep an index mapping portions of data to hash values in memory in order to detect duplicate data without paying the performance penalty of disk paging. The index size is proportional to the volume of unique data, so placing the entire index into RAM is not cost effective with a deduplication ratio below 45%. HANDS reduces the amount of in-memory index storage required by up to 99% while still achieving between 30% and 90% of the deduplication a full memory-resident index provides, making primary deduplication cost effective in workloads with deduplication rates as low as 8%. HANDS is a framework that dynamically pre-fetches fingerprints from disk into memory cache according to working sets statistically derived from access patterns. We use a simple neighborhood grouping as our statistical technique to demonstrate the effectiveness of our approach. HANDS is modular and requires only spatio-temporal data, making it suitable for a wide range of storage systems without the need to modify host file systems.

HANDS: A heuristically arranged non-backup in-line deduplication system

Digital archives are growing rapidly, necessitating stronger reliability measures than RAID to avoid data loss from device failure. Mirroring, a popular solution, is too expensive over time. We present a compromise solution that uses multi-level redundancy coding to reduce the probability of data loss from multiple simultaneous device failures. This approach handles small-scale failures of one or two devices efficiently while still allowing the system to survive rare-event, larger-scale failures of four or more devices. In our approach, each disk is split into a set of fixed size disklets which are used to construct reliability stripes. To protect against rare event failures, reliability stripes are grouped into larger super-groups, each of which has a corresponding super-parity; super-parity is only used to recover data when disk failures overwhelm the redundancy in a single reliability stripe. Super-parity can be stored on a variety of devices such as NV-RAM and always-on disks to offset write bottlenecks while still keeping the number of active devices low. Our calculations of failure probabilities show that adding super-parity allows our system to absorb many more disk failures without data loss. Through discrete event simulation, we found that adding super-groups has a significant impact on mean time to data loss and that rebuilds are slow but not unmanageable. Finally, we showed that robustness against rare events can be achieved for a fraction of total system cost.

Avani Wildani

Papers

Estimating PM2.5 Concentrations in the Conterminous United States Using the Random Forest Approach.

Incorporating Low-Cost Sensor Measurements into High-Resolution PM2.5 Modeling at a Large Spatial Scale.

Impacts of snow and cloud covers on satellite-derived PM2.5 levels.

HANDS: A heuristically arranged non-backup in-line deduplication system

Protecting against rare event failures in archival systems