Basics of qualitative research: Grounded theory procedures and techniques

Data Mining Practical Machine Learning Tools and Techniques

A review of assessing the accuracy of classifications of remotely sensed data

Remote sensing imagery needs to be converted into tangible information which can be utilised in conjunction with other data sets, often within widely used Geographic Information Systems (GIS). As long as pixel sizes remained typically coarser than, or at the best, similar in size to the objects of interest, emphasis was placed on per-pixel analysis, or even sub-pixel analysis for this conversion, but with increasing spatial resolutions alternative paths have been followed, aimed at deriving objects that are made up of several pixels. This paper gives an overview of the development of object based methods, which aim to delineate readily usable objects from imagery while at the same time combining image processing and GIS functionalities in order to utilize spectral and contextual information in an integrative way. The most common approach used for building objects is image segmentation, which dates back to the 1970s. Around the year 2000 GIS and image processing started to grow together rapidly through object based image analysis (OBIA - or GEOBIA for geospatial object based image analysis). In contrast to typical Landsat resolutions, high resolution images support several scales within their images. Through a comprehensive literature review several thousand abstracts have been screened, and more than 820 OBIA-related articles comprising 145 journal papers, 84 book chapters and nearly 600 conference papers, are analysed in detail. It becomes evident that the first years of the OBIA/GEOBIA developments were characterised by the dominance of ‘grey’ literature, but that the number of peer-reviewed journal articles has increased sharply over the last four to five years. The pixel paradigm is beginning to show cracks and the OBIA methods are making considerable progress towards a spatially explicit information extraction workflow, such as is required for spatial planning as well as for many monitoring programmes.

Object based image analysis for remote sensing

http://www.geog.ucl.ac.uk/~mdisney/teaching/PPRS/papers/foody.pdf

Status of land cover classification accuracy assessment

Introduction Why Accuracy Assessment? Overview Historical Review Aerial Photography Digital Assessments Data Collection Considerations Classification Scheme Statistical Considerations Data Distribution Randomness Spatial Autocorrelation Sample Size Sampling Scheme Sample Unit Reference Data Collection Basic Collection Forms Basic Analysis Techniques Non-Site Specific Assessments Site Specific Assessments Area Estimation/Correction Practicals Impact of Sample Design on Cost Recommendations for Collecting Reference Data ASources of Variation in Reference Data Photo Interpretation vs. Ground Visitation Interpreter Variability Observations vs. Measurements What is Correct? Labeling Map vs. Labeling the Reference Data Qualitative vs. Quantitative Analysis Local vs. Regional vs. Global Assessments Advanced Topics Beyond the Error Matrix Modifying the Error Matrix Fuzzy Set Theory Measuring Variability Complex Data Sets Change Detection Multi-Layer Assessments California Hardwood Rangeland Monitoring Project Case Study Balancing Statistical Validity with Practical Reality Bibliography

Assessing the accuracy of remotely sensed data : principles and practices

Accuracy assessment: a user's perspective

Discrete multivariate analysis techniques have been used to evaluate the accuracy of land-cover classifications from Landsat digital image y . Error matrices or contingency tables were taken from the literature and then analyzed using three techniques. The first technique permitted direct comparison of corresponding cell values in different matrices by \"nomnalizing\" each matrix through a process called \"iterative proportional fitting.\" The second technique provided a method of testing for signijicant differences between error matrices which vary by only a single variable. The third technique allowed for multivariable comparisons between matrices to be made and is the most powerful of the techniques. It was concluded that these techniques could help researchers better evaluate variables or factors affecting class$cation accuracy. scope of this paper to present all the theoretical and practical details of the statistical analysis involved here. However, it is hoped that even those users with little to no statistical background will be able to see the usefulness of these techniques from this presentation. The method of accuracy assessment described * Presently employed by Technicolor Government Services, Inc., Denver, Colorado. PHOTOGRAMMETRIC ENGINEERING AND REMOTE SENSING, Vol. 49, No. 12, December 1983, pp. 1671-1678. The most common way to represent the accuracy of a Landsat classification is in the form of an error matrix or contingency table (e.g., Card, 1982; Mead and Meyer, 1977; Hoffer, 1975). An error matrix is a square array of numbers set out in rows and columns which express the number of pixels assigned as a particular land-cover type relative to the actual land cover as verified in the field or from interpreted aerial photographs. The columns usually represent the reference data (i.e., assumed correct) and 0099-1112/83/4912-1671$02.25/0 O 1983 American Society of Photogrammetry PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING, 1983 the rows indicate the computer assigned land-cover category (i.e., Landsat data). This form of expressing accuracy as an error matrix is an effective way to evaluate both errors of inclusion (commission errors) and errors of exclusion (omission errors) present in the classification. Also, the error matrix allows the analyst to determine the performance for individual categories as well as for the overall classification (Hoffer and Fleming, 1978). In the ideal situation, all the non-major diagonal elements of the error matrix would be zero, indicating that no pixel had been misclassified. Once the error matrix has been generated, a very simple procedure can be used to determine the overall accuracy. Because the values on the major diagonal represent those pixels that have been correctly classified, these values are summed up and divided by the total number of pixels classified. This number is then the overall performance accuracy of an error matrix, and is the most common use of the error matrix in accuracy assessment. Until recently, this measure of overall performance accuracy was the extent of most accuracy assessments. However, additional statistical techniques are now being used to further assess classification accuracy. These methods can be divided into two groups: analysis of variance and discrete multivariate analysis (often called contingency table analysis). Analysis of variance makes use of only the diagonal elements in the error matrix. Also, the technique requires that the data be normally distributed. As previously mentioned, classification data are discrete and multinomially distributed (each category is binomially distributed). The diagonal elements of the error matrix can be converted to a normal distribution using various transformations (Snedecor and Cochran, 1976). However, another assumption of analysis of variance is that the categories in the error matrix are independent. This assumption is often not met in remotely sensed data because of the confusion between categories. For additional details and examples of this technique, see Rosenfield (1982). Discrete multivariate analysis, on the other hand, does not assume that the categories are independent nor does it require any transformation of the data. Instead, these techniques are designed specifically to deal with categorical data. Discrete multivariate analysis also uses the entire error matrix and not just the diagonal elements. As suggested by Card (1982), \"contingency table analysis is the most natural framework for accuracy assessment, both for the convenient display of empirical results and for the ease of statistical analysis.\" Three different methods of comparing error matrices using discrete multivariate analysis techniques were evaluated in this study. The first method allows for direct comparison of error matrices through a process called normalization. The second method computes a measure of agreement between error matrices which can be used to test if the matrices are significantly daerent . The third method provides for the simultaneous examination of all factors affecting the classification. The first comparison procedure (Bishop et al., 1975) allows corresponding cell values in ddferent error matrices to be directly compared. This comparison is made possible by a standardizing process called normalization. Normalization of an error matrix is performed by a procedure called \"iterative proportional fitting.\" The rows and columns of a matrix are successively balanced until each row and each column adds up to a given value (say 1.0). This process forces each cell value to be influenced by all the other cells values in its corresponding row and column. Each cell is then a combination of ground truth and computer classification and is representative of both omission and commission errors for that land-cover category. Prior to the normalization procedure, comparison of corresponding cell values in different matrices was only possible if the matrices had the same s a m ~ l e size. Even then. the cell value mav have been misleading because errors of omission and commission were ignored. However, due to the normalization procedure, the corresponding cell values of two or more error matrices can now be compared directly without regard for differences in sample size and including omission and commission errors. Although there is no test for significance between corresponding cell values, direct comparison can provide a relative measure of which is better because all columns and rows in the matrices are required to sum to a certain marginal. The normalization procedure converges to a unique set of maximum likelihood estimates and as such is the most appropriate algorithm to use in this case (Fienberg, 1970). However, an assumption made by this procedure is that all cells are of equal weight or importance. This assumption is not always valid in remotely sensed data. However, it is possible to modify the fitting procedure to deal with categories of unequal importance (Bishop et al. , 1975). In either case, this method does provide a way of eliminating the effect of sample size while incorporating omission and commission errors into the accuracy measurement. The second method of comparison examined here is a procedure that tests if the overall agreement in two separate error matrices is significantly different. A measure of overall agreement is computed for each matrix based on the difference between the actual agreement of the classification (i.e., agreement between computer classification and reference data as indicated by the diagonal elements) and the chance agreement which is indicated by the product of the row and column margiqals. This measure of agreement, called KHAT (i.e., K ) , is calculated by DISCRETE MULTIVARIATE ANALYSIS TECHNIQUES where r is the number of rows in the matrix, xii is the number of observations in row i and column i (i.e., the ith diagonal element), xi+ and x + ~ are the marginal totals of row i and column i, respectively, and N is the total number of observations (Bishop et al., 1975). Notice that the numerator of this equation is similar to the observed minus the expected calculation performed in a chi-square analysis. A KHAT value is coinputed for each matrix and is a measure of how well the classification agrees with the reference data (i.e., a measure of overall accuracy). Confidence intervals can be calculated for KHAT using the approximate large sample variance (Bishop et al . , 1975, p. 396) 1 8, (1 01) + 2(1 01)(20102 03) s ( K ) = N (1 8,)' (1 0 2

Assessing Landsat classification accuracy using discrete multivariate analysis statistical techniques.

A method has been developed to quantitatively test the degree of similarity between photointerpreters andlor photointerpretation variables, such as filmlfilter type, season, and scale. This method involves giving each photointerpreter the same set of photos to interpret, and then a similarity matrix is generated for each interpreter by comparing the interpretation results to the actual ground cover. It is assumed that the interpretation has been sampled in such a way as to assure that the resulting similarity matrix is representative of that interpretation. The similarity matrix is analyzed using a computer program called KAPPA, which implements a discrete multivariate analysis technique to determine if there is a significant dgference between matrices. This technique allows the comparison of individual interpreters, a test for photointerpreter consistency, and the comparison of photointerpretation variables. method of quantifying photointerpretation results consistency in his interpretation skill. Lastly, it and gives a statistical method by which interpretmay also be important to determine if varying ers and other photointerpretation variables such as types of photography (fildfilter combinations) or fildfilter types can be compared. seasons (spring versus summer) of ~hotography The procedure proposed in this paper can test result in significantly different identifications. for significant differences between photointerPROCEDURE * Presented, in part, at the 1981 Annual Convention of the American Society of Photogrammetry, WashingThe first step in quantifying photointerpretation ton, D.C. results is to cellularize the photographs. There are PHOTOCRAMMETRIC ENG NEERING AND REMOTE SENSING, Vol. 49, No. 1, January 1983, pp. 69-74. 0099-11 12/83/4901-69$02.25/0 @ 1983 American Society of Photogrammetry PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING, 1983 many ways that this may be accomplished. The identifications or delineations made by the interpreter can be digitized, and then a given cell size can be used to assign the majority land-usel land-cover category to each cell. A dot grid could also be used in which each dot defines the sample point. Obviously, the results of the assignment to categories will vary with different cell sizes or dots per square inch. Therefore, it is important to choose a cell size that is indicative of the objectives of the photointerpretation (i.e., the more detail required, the smaller the cell size). The minimum mapping unit used in the photointerpretation is a good choice for the cell size. Once a grid system has been placed over the photo and each cell has been assigned to some category, then each cell is compared with its corresponding cell from another interpretation. If one of these interpretations is assumed to be correct (reference data), then comparison of the two sets of spatially defined cells yields a measure of \"photointerpretation accuracy.\" This comparison is usually expressed in the form of an error matrix. It will be assumed in this paper that the error matrices generated are representative of the actual photointerpretation results. This means that an appropriate sampling scheme was chosen and an adequate number of samples were taken when compiling the matrix. ~ e c a u s e the technique proposed here analyzes the error matrices, it is necessary that this assumption hold for any results to be meaningful. An error matrix is a square array set out in rows and columns which expresses the number of cells assigned to a particular land-cover type (photointerpretation) relative to the actual land cover (reference data). The columns represent the reference data while the rows indicate the photointerpreter assigned land-cover type (Table 1). The numbers in the error matrix are the actual tallies compiled by comparing the photointerpretation with the actual land-cover type on a cell by cell basis. As can be seen in Table 1, all correct classifications are indicated on the major diagonal of the error matrix. For example, in this case 16 cells were correctly classified as forest. Also note that the offdiagonal elements indicate misclassification, which is a combination of omission and commission error. The specific method used to generate an error matrix is dependent upon what information is desired. If the similarity between two or more photointerpreters is to be determined, then each interpreter is given the same aerial photographs to interpret. A similarity matrix is tabulated for each interpreter by comparing his interpretation with a reference data set (assumed correct). If the test involves determining the consistency over time for a single interpreter, then a representative sample of the photos is selected for interpretation at the beginning of the project. At some later date, the remainder of the photos are interpreted and the two similarity matrices (Time I versus Reference Data and Time I1 versus Reference Data) are compared. Finally, if it is desired to measure the similarity of identifications made on different types of photography, a separate interpretation is performed on the same area for each set of photos by each interpreter and the appropriate similarity matrices are generated. Once the similarity matrices have been generated, a discrete multivariate analysis technique (Bishop et al., 1975) is used to test the agreement between matrices. This technique is appropriate because the data are discrete and multinomially distributed. Parametric statistical techniques such as analysis of variance assume that the data are continuous and normally distributed. Chi square analysis is possible except that problems arise when cell values in the matrix equal zero. This problem does not affect the discrete multivariate analysis technique. The statistic used to test the agreement between similarity matrices is called (KHAT). KHAT is the TABLE 1. EXAMPLE ERROR MATRIX FOR

Russell G. Congalton

Papers

A review of assessing the accuracy of classifications of remotely sensed data

Assessing the accuracy of remotely sensed data : principles and practices

Accuracy assessment: a user's perspective

Assessing Landsat classification accuracy using discrete multivariate analysis statistical techniques.

A Quantitative Method to Test for Consistency and Correctness in Photointerpretation