This paper surveys theoretical and practical issues associated with a particular type of information retrieval problem, namely that where the information need is pictorial. The paper is contextualised by the notion of a visually stimulated society, in which the ease of record creation and transmission in the visual medium is contrasted with the difficulty of gaining effective subject access to the world's stores of such records. The technological developments which, in casting the visual image in electronic form, have contributed so significantly to its availability are reviewed briefly, as a prelude to the main thrust of the paper. Concentrating on still and moving pictorial forms of the visual image, the paper dwells on issues related to the subject indexing of pictorial material and discusses four models of pictorial information retrieval corresponding with permutations of the verbal and visual modes for the representation of picture content and of information need.

Progress in documentation pictorial information retrieval

The paper reviews six recent efforts to better understand performance measurements on information retrieval (IR) systems within the framework of the Text REtrieval Conferences (TREC): analysis of variance, cluster analyses, rank correlations, beadplots, multidimensional scaling, and item response analysis. None of this work has yielded any substantial new insights. Prospects that additional work along these lines will yield more interesting results vary but are in general not promising. Some suggestions are made for paying greater attention to richer descriptions of IR system behavior but within smaller, better controlled settings.

Blind Men and Elephants: Six Approaches to TREC data

This study asks, how well does an item response theory (IRT) based computerized adaptive NEO PI-R work? To explore this question, real-data simulations (N = 1,059) were used to evaluate a maximum information item selection computerized adaptive test (CAT) algorithm. Findings indicated satisfactory recovery of full-scale facet scores with the administration of around four items per facet scale. Thus, the NEO PI-R could be reduced in half with little loss in precision by CAT administration. However, results also indicated that the CAT algorithm was not necessary. We found that for many scales, administering the "best" four items per facet scale would have produced similar results. In the conclusion, we discuss the future of computerized personality assessment and describe the role IRT methods might play in such assessments.

Computerization and Adaptive Administration of the NEO PI-R

A simulation study was conducted to investigate the application of expected a posteriori (EAP) trait estimation in computerized adaptive tests (CAT) based on the partial credit model and compare it with maximum likelihood trait estimation (MLE). The performance of EAP was evaluated under different conditions: the number of quadrature points (10, 20,40, and 80) and the type of prior distribution (normal and uniform). The relative performance of MLE and the EAP estimation methods was assessed under two distributional forms of the latent trait (normal and negatively skewed). Results showed that, regardless of the latent trait distribution, MLE and EAP with a normal prior or a uniform prior using either 20, 40, or 80 quadrature points provided relatively accurate estimation in CAT based on the partial credit model. Also, increasing the number of quadrature points from 20 to 80 did not increase the accuracy of EAP estimation.

A Comparison of Maximum Likelihood Estimation and Expected a Posteriori Estimation in CAT Using the Partial Credit Model

Multiple similarity measures for five TREC topic-document sets from the LDC TREC Collection Disk 1 are derived from the full text of documents. Each measure on each set is scaled using SAS MDS under ordinal, interval, and MLE assumptions. The resulting 75 permutations are ploted. It is suggested that cosine-vector and overlap measures for similarity appear to recover optimal data relationships among the documents of the five sets. MLE assumptions appear to be required to model the data adequately.

Images of similarity: a visual exploration of optimal similarity metrics and scaling properties of TREC topic-document sets

A simulation study was conducted to investigate the effect of population distribution on maximum likelihood estimation (MLE) and expected a posteriori estimation (EAP) in computerized adaptive testing (CAT) based on Andrich's rating scale model. Comparisons were made among MLE and EAP with a normal prior distribution and EAP with a uniform prior distribution within two data sets: one generated using a normal trait distribution and the other using a negatively skewed trait distribution. Descriptive statistics, correlations, scattergrams, and accuracy indices were used to compare the different methods of trait estimation. EAP estimation with a normal prior or uniformprior yielded results similar to those obtained with MLE, even though the prior did not match the underlying trait distribution. An additional simulation study based on real data suggested that more work is needed to determine the optimal number of quadrature points for EAP in CAT based on the rating scale model. The choice between MLE and EAP f...

The Effect of Population Distribution and Method of Theta Estimation on Computerized Adaptive Testing (CAT) Using the Rating Scale Model

Although numerous new document visualization tools are emerging throughout academia and industry, reliable test data for such tools has not yet been established. This paper explores the applicability of the TREC Information Retrieval Test Collection for this purpose using commonly available data and statistical methods.

Visualization and scaling of TREC topic document sets

The method described permits visual analysis of information retrieval experiment results in classic control and treatment group protocols. It is an additional analysis technique for the evaluation of IR procedures that may be conducted within the rich terrain of human visual acuity, supported by two well-known statistical measures.

Shape recovery: a visual method for evaluation of information retrieval experiments

Human beings judge images by complex mental processes, whereas computing machines extract features. By reducing scaled human judgments and machine extracted features to a common metric space and fitting them by regression, the judgments of human experts rendered on a sample of images may be imposed on an image population to provide automatic classification.

/pdf/a-new-machine-classification-method-applied-to-human-4ajfyb01cp.pdf

A new machine classification method applied to human peripheral blood leukocytes

IRTINFO is a collection of SAS macros that compute item and test information for the graded response model (Samejima, 1969), the partial credit model (Masters, 1982), the generalized partial credit model (Muraki, 1992), the rating scale model (Andrich, 1978), the successive intervals model (Rost, 1988), and the three-parameter logistic model (~irnbaurn, 1968). Information is computed at each level of 0 in a range of 0 values specified by the user. The macros can handle items with differing numbers of categories when

Steven J. Fitzpatrick

Papers

The Effect of Population Distribution and Method of Theta Estimation on Computerized Adaptive Testing (CAT) Using the Rating Scale Model

Visualization and scaling of TREC topic document sets

Shape recovery: a visual method for evaluation of information retrieval experiments

A new machine classification method applied to human peripheral blood leukocytes

IRTINFO: A SAS Macro Program to Compute Item and Test Information: