25. Multiple Imputation for Nonresponse in Surveys. By D. B. Rubin. ISBN 0 471 08705 X. Wiley, Chichester, 1987. 258 pp. £30.25.

Multiple Imputation for Nonresponse in Surveys

All digital data contain error and many are uncertain. Digital models of elevation surfaces consist of files containing large numbers of measurements representing the height of the surface of the earth, and therefore a proportion of those measurements are very likely to be subject to some level of error and uncertainty. The collection and handling of such data and their associated uncertainties has been a subject of considerable research, which has focused largely upon the description of the effects of interpolation and resolution uncertainties, as well as modelling the occurrence of errors. However, digital models of elevation derived from new technologies employing active methods of laser and radar ranging are becoming more widespread, and past research will need to be re-evaluated in the near future to accommodate such new data products. In this paper we review the source and nature of errors in digital models of elevation, and in the derivatives of such models. We examine the correction of errors and assessment of fitness for use, and finally we identify some priorities for future research.

http://journals.sagepub.com/doi/pdf/10.1191/0309133306pp492ra

Causes and consequences of error in digital elevation models

Statistics for spatial data: by Noel Cressie, 1991, John Wiley & Sons, New York, 900 p., ISBN 0-471-84336-9, US $89.95

Spatial data quality is a paramount concern in all GIS applications. Existing spatial data accuracy standards, including the National Standard for Spatial Data Accuracy (NSSDA) used in the United States, commonly assume the positional error of spatial data is normally distributed. This research has characterized the distribution of the positional error in four types of spatial data: GPS locations, street geocoding, TIGER roads, and LIDAR elevation data. The positional error in GPS locations can be approximated with a Rayleigh distribution, the positional error in street geocoding and TIGER roads can be approximated with a log-normal distribution, and the positional error in LIDAR elevation data can be approximated with a normal distribution of the original vertical error values after removal of a small number of outliers. For all four data types considered, however, these solutions are only approximations, and some evidence of non-stationary behavior resulting in lack of normality was observed in all four datasets. Monte-Carlo simulation of the robustness of accuracy statistics revealed that the conventional 100% Root Mean Square Error (RMSE) statistic is not reliable for non-normal distributions. Some degree of data trimming is recommended through the use of 90% and 95% RMSE statistics. Percentiles, however, are not very robust as single positional accuracy statistics. The non-normal distribution of positional errors in spatial data has implications for spatial data accuracy standards and error propagation modeling. Specific recommendations are formulated for revisions of the NSSDA.

/pdf/positional-accuracy-of-spatial-data-non-normal-distributions-3gw6zj5anc.pdf

Positional Accuracy of Spatial Data: Non-Normal Distributions and a Critique of the National Standard for Spatial Data Accuracy

Publisher Summary This chapter gives guidance on how to prepare elevation data for geomorphometric analysis. It outlines some common errors in raw digital elevation model (DEM) data sources and then suggests approaches for systematically improving the quality of DEMs. It starts with height samples and ends with the final DEMs used for geomorphometric analysis. More precisely, the chapter describes both simple and more advanced algorithms that can be used to reduce systematic and random errors, and enrich the quality of DEMs by incorporating auxiliary information on land cover and the hydrological properties of an area. These algorithms are implemented and described further in the software packages. The nature of DEM-preprocessing algorithms very much depends on the type1 of input data. For this reason, not all algorithms are applicable to raw DEMs. From all the approaches to preprocessing DEMs to reduce errors, three different groups are distinguished: The empirical methods, the filtering methods, and the simulation methods.

Chapter 4 Preparation of DEMs for Geomorphometric Analysis

http://www.thedigitalmap.com/~carlos/papers/rep07_1/DiscoveringKey.pdf

Discovering key meteorological variables in atmospheric corrosion through an artificial neural network model

The widespread availability of powerful desktop computers, easy-to-use software tools and geographic datasets have raised the quality problem of input data to be a crucial one. Even though accuracy has been a concern in every serious application, there are no general tools for its improvement. Some particular ones exist, however, and some results are presented here for a particular case of quantitative raster data: Digital Elevation Models (DEM). Two procedures designed to detect anomalous values (also named gross errors, outliers or blunders) in DEMs, but valid also for other quantitative raster datasets, were tested. A DEM with elevations varying from 181 to 1044 m derived from SPOT data has been used as a contaminated sample, while a manually derived DEM obtained from aerial photogrammetry was regarded as the ground truth to allow a direct performance comparison for the methods with real errors. It is assumed that a “better” value can be measured or obtained through some methodology once an outlier location is suggested. The options are different depending upon the user (DEM producers might go to the original data and make another reading, while end users might use interpolation). Both choices were considered in this experiment. Preliminary results show that for the available dataset, the accuracy might be improved to some extent with very little effort. Effort is defined here as the percentage of points suggested by the methodology in relation with its total number: thus 100 per cent effort implies that all points have been checked. The method proposed by Lopez (1997) gave poor results, because it has been designed for errors with low spatial autocorrelation (which is not the case here). A modified version was then designed and compared with the method proposed by Felicisimo (1994). The three procedures can be applied both for error detection during DEM generation and by end users, and they might be of use for other quantitative raster data. The choice of the best methodology is different depending on the effort involved. The conclusions have been derived for a photogrammetrically obtained DEM; other production procedures might lead to different results.

/pdf/improving-the-elevation-accuracy-of-digital-elevation-models-2x66vyh3mq.pdf

Improving the Elevation Accuracy of Digital Elevation Models: A Comparison of Some Error Detection Procedures

We describe how the graded minimal resolution of certain semigroup algebras is related to the combinatorics of some simplicial complexes We obtain characterizations of the Cohen-Macaulay and Gorenstein conditions The Cohen- Macaulay type is computed from combinatorics As an application, we compute explicitly the graded minimal resolution of monomial both affine and simplicial projective surfaces

Combinatorics of syzygies for semigroup algebras

The techniques employed in the treatment of an hourly surface wind database during the development and calibration phases of an objective wind field interpolator model are presented. The model itself has been applied to estimate the regional wind energy resource creating a layer in a GIS environment. The outlier detection phase is presented in a companion paper, and here the different techniques applied in order to imputate the missing values are described. The comparative results obtained with an hourly dataset of 15 years long are also presented. Two different problems have been simulated numerically: systematic missing values (i.e. at fixed hours) and non systematic ones. Five different criteria were applied: imputation with the historical mean value; linear time interpolation within single station records; optimum interpolation (kriging) and the two newly developed Penalty Of the Principal Scores and linear Time Interpolation of the Principal Scores which considers all station records in a multivariate fashion; they prove to be the most accurate for this particular wind dataset. There is also some evidence of oversampling in time.

/pdf/a-new-technique-for-imputation-of-multivariate-time-series-yjpd2zdn1g.pdf

A new technique for imputation of multivariate time series: application to an hourly wind dataset

In order to increase the overall performance of distributed parallel programs running in a network of non-dedicated workstations, we have researched methods for improving load balancing in loosely coupled heterogeneous distributed systems. Current software designed to handle distributed applications does not focus on the problem of forecasting the computers future load. The software only dispatches the tasks assigning them either to an idle CPU (in dedicated networks) or to the lowest loaded one (in non-dedicated networks). Our approach tries to improve the standard dispatching strategies provided by both parallel languages and libraries, by implementing new dispatching criteria. It will choose the most suitable computer after forecasting the load of the individual machines based on current and historical data. Existing applications could take advantage of this new service with no extra changes but a recompilation. A fair comparison between different dispatching algorithms could only be done if they run over the same external network load conditions. In order to do so, a tool to arbitrarily replicate historical observations of load parameters while running the different strategies was developed. In this environment, the new algorithms are being tested and compared to verify the improvement over the dispatching strategy already available. The overall performance of the system was tested with in-house developed numerical models. The project reported here is connected with other efforts at CeCal devoted to make it easier for scientists and developers to gain advantage of parallel computing techniques using low cost components. 1 This research was supported in part by CSIC 2 IEEE member 3 Centro de Cálculo. Julio Herrera y Reissig 565 5to piso. CP 11.300. Montevideo. Uruguay http://www.fing.edu.uy/cecal Introduction Parallel architectures are nowadays wide spread. With the growth of node power and the increases of network bandwidth, present networks can be used as powerful parallel environments. This kind of hardware support for distributed processing can be found in a growing number of companies and universities. However, software support for efficiently use this kind of architectures is still not good enough. There is almost no visual environment for developing distributed systems, and the ones that exist are extremely expensive. On the other hand, they are (in many senses) under-developed. In addition, they only have rudimentary tools to cope with unexpected overload in a node. This usually leads to a significant delay in the computation time. CeCal [1] is concerned about this topic, and has many concluded ([2][3][4][5]) and ongoing ([6][7][8]) research projects. This project proposes to improve PVM [9] dispatching algorithms in order to use historical information about the use of the computers that compose the virtual machine. New algorithms based in neural networks techniques and traditional statistical methods (ARMA, ARIMA) were used to predict future usage of individual workstations based on historical usage. In order to compare two forecasting algorithms under the same workload environment, two instances of the same program with just the dispatching criteria changed should be run. However, this executions should run in a background load environment as similar as possible. This paper describes the work done on this direction, both about the development of new dispatching algorithms and in the infrastructure needed to fairly compare them against the standard ones (network load gatherer and replicator). These new dispatching algorithms were tested using UCLA ́s Global Climate Model [10] and a Shallow Water Model. [11] Load Balancing and new dispatching criteria It has been proved that load balancing can be considered as the most relevant performance enhancement in a multi-user network environment. When many tasks are used for solving some problem, a single delay produces the whole system be delayed, with the other tasks waiting for the delayed one at synchronizing points. Load balancing techniques allow distributed applications to take care about this problem. Standard load balancing strategies for distributed applications propose choosing the most appropriate workstation of the virtual machine. The most appropriate workstation means the lowest loaded machine at task spawning time. All the history of workstations load is ignored and only instant load is considered. Our project proposed and implemented two original methods for improving load balancing of distributed applications. In both cases, load balancing will be considered before tasks are started, by forecasting load in the target computer. Figure 1 By considering statistical information, load balancing could be improved. Our project will include services in the PVM kernel for gathering virtual machine workstations load information, and using it at task spawning time in order to forecast future workstation load and correctly define the "most appropriated workstation" choice. Different time series forecasting methods were compared; predictors were implemented and a framework was developed in order to test and compare improved vs. traditional dispatch. Determination of Load Meaningful Parameters There exists many and different workload that can be considered for load balancing improvement: number of tasks in the run queue, size of free available memory, 1-min load average, amount of free CPU time, and so on. In order to improve PVM dispatching routine we have to predict, in some way, future load of the workstations that compose the virtual machine. This prediction will be based in some combination of the selected workload parameter. This value is usually called the load index. In order to compare our new dispatching strategy, we need to run the same PVM program in the same virtual machine load environment with both the old and the new dispatching routines. So, we must be able to generate a workload artificially, for arbitrarily replicate historical observations of load parameters while applying the different strategies. The framework developed for testing and comparing different task dispatching algorithms (the Load Replicator) will be discussed below. We needed support for replicating, at any time, a given network load situation. Only with this artificially generated background workload, one could fairly compare different solutions for the dispatching of a distributed program. The input is, for N workstations, a time series of the load observed for some representative parameters, and the output will be each workstation loaded with the prescribed load. UNIX systems provide at many levels of detail information about system usage [12]. The load information we used is the one provided by the rstat service, which gathers information about CPU usage, local (non NFS) disk usage, paging and swapping, interrupts and context switches, network usage and collisions. Some of them are graphically represented in figure 1. Looking for simplicity and transparent processing, we discarded some parameters that have no effect in global load patterns. For example, kernel operations like context switching were ignored because they are extremely hard to trace and replicate, and they do not affect global workstation performance [13]. Also, parameters related with misconfiguration problems (for example, if excessive swapping and paging is observed, possible exists a memory leak problem) were ignored. On the other hand, parameters that hardly change or are not meaningful over the time (like collisions) also were neglected. The identification of suitable load indexes is not a new problem [14], and is well known that simple load indexes are particularly effective. Kunz [15] found that the most effective of the indexes we have mentioned above is the CPU queue length. However, due to the specific topic we are managing (network parallel computing) where network traffic is a bottleneck and distributed processes usually make great use of local disk, we considered that this two indexes should not be neglected. Artificial Replication of Network Load Environment Under the assumptions described, we have replicated just CPU, disk and network usage for each workstation. Replication is not a new research area, and it is needed for testing purposes in many computer science areas. Workload replicators for measuring performance of different file system implementations were studied in [16][17]. Also replicators for real-time system applications were developed that work at the kernel level, obtaining excellent results for very short periods of time [14]. As our purpose is to replicate workloads in a complete network with many parameters of each workstation, we don’t need excessive precision in the replication at the microsecond scale, but we expect acceptable results for the long periods (like hours). So we decided to work at the process level. In this paper we will focus only on CPU replication; the other parameter replication are exactly the same, just changing the loader subprocess. The procedure is very simple: if historical load is bigger than actual load, then we do hard work in non-cacheable tasks; if not, we just wait. Periodically we compare both loads and we act in consequence. Figure 3 – Replicated CPU load Figure 2 – Historical CPU load Despite its simplicity, this strategy works fine. We can visually compare the historical load (figure 2) and the replicated one (figure 3). Vertical axis shows the percent of CPU used during the measured interval, with increments in the range 1-100 per second. Horizontal axis indicates historical measurements, taken every 60 seconds in these experiments. These units are used in all experiments shown in this paper, unless is specified. Errors obtained for CPU replication are shown in figure 4. Working with a 60 seconds interval, average deviation was of 0,97 while the maximum deviation was 5. The vertical axis units are those provided by rstat. In order to

/pdf/towards-proactive-network-load-management-for-distributed-4pujrg29i9.pdf

Carlos López

Papers

Discovering key meteorological variables in atmospheric corrosion through an artificial neural network model

Improving the Elevation Accuracy of Digital Elevation Models: A Comparison of Some Error Detection Procedures

Combinatorics of syzygies for semigroup algebras

A new technique for imputation of multivariate time series: application to an hourly wind dataset

Towards Proactive Network Load Management for Distributed Parallel Programming.