Showing papers in "Journal of The Royal Statistical Society Series A-statistics in Society in 2005"

Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators

Abstract: Summary. Composite indicators are increasingly used for bench-marking countries’ performances. Yet doubts are often raised about the robustness of the resulting countries’ rankings and about the significance of the associated policy message. We propose the use of uncertainty analysis and sensitivity analysis to gain useful insights during the process of building composite indicators, including a contribution to the indicators’ definition of quality and an assessment of the reliability of countries’ rankings. We discuss to what extent the use of uncertainty and sensitivity analysis may increase transparency or make policy inference more defensible by applying the methodology to a known composite indicator: the United Nations's technology achievement index.

Multiple-bias modelling for analysis of observational data

Abstract: Summary. Conventional analytic results do not reflect any source of uncertainty other than random error, and as a result readers must rely on informal judgments regarding the effect of possible biases. When standard errors are small these judgments often fail to capture sources of uncertainty and their interactions adequately. Multiple-bias models provide alternatives that allow one systematically to integrate major sources of uncertainty, and thus to provide better input to research planning and policy analysis. Typically, the bias parameters in the model are not identified by the analysis data and so the results depend completely on priors for those parameters. A Bayesian analysis is then natural, but several alternatives based on sensitivity analysis have appeared in the risk assessment and epidemiologic literature. Under some circumstances these methods approximate a Bayesian analysis and can be modified to do so even better. These points are illustrated with a pooled analysis of case–control studies of residential magnetic field exposure and childhood leukaemia, which highlights the diminishing value of conventional studies conducted after the early 1990s. It is argued that multiple-bias modelling should become part of the core training of anyone who will be entrusted with the analysis of observational data, and should become standard procedure when random error is not the only important source of uncertainty (as in meta-analysis and pooled analysis).

All of Statistics: a Concise Course in Statistical Inference

Abstract: Taken literally, the title \"All of Statistics\" is an exaggeration. But in spirit, the title is apt, as the book does cover a much broader range of topics than a typical introductory book on mathematical statistics. This book is for people who want to learn probability and statistics quickly. It is suitable for graduate or advanced undergraduate students in computer science, mathematics, statistics, and related disciplines. The book includes modern topics like nonparametric curve estimation, bootstrapping, and classification, topics that are usually relegated to follow-up courses. The reader is presumed to know calculus and a little linear algebra. No previous knowledge of probability and statistics is required. Statistics, data mining, and machine learning are all concerned with collecting and analysing data.

Variance partitioning in multilevel logistic models that exhibit overdispersion

Abstract: Summary. A common application of multilevel models is to apportion the variance in the response according to the different levels of the data. Whereas partitioning variances is straightforward in models with a continuous response variable with a normal error distribution at each level, the extension of this partitioning to models with binary responses or to proportions or counts is less obvious. We describe methodology due to Goldstein and co-workers for apportioning variance that is attributable to higher levels in multilevel binomial logistic models. This partitioning they referred to as the variance partition coefficient. We consider extending the variance partition coefficient concept to data sets when the response is a proportion and where the binomial assumption may not be appropriate owing to overdispersion in the response variable. Using the literacy data from the 1991 Indian census we estimate simple and complex variance partition coefficients at multiple levels of geography in models with significant overdispersion and thereby establish the relative importance of different geographic levels that influence educational disparities in India.

Performance indicators: good, bad, and ugly

Abstract: Summary. A striking feature of UK public services in the 1990s was the rise of performance monitoring (PM), which records, analyses and publishes data in order to give the public a better idea of how Government policies change the public services and to improve their effectiveness. PM done well is broadly productive for those concerned. Done badly, it can be very costly and not merely ineffective but harmful and indeed destructive. Performance indicators (PIs) for the public services have typically been designed to assess the impact of Government policies on those services, or to identify well performing or underperforming institutions and public servants. PIs’ third role, which is the public accountability of Ministers for their stewardship of the public services, deserves equal recognition. Hence, Government is both monitoring the public services and being monitored by PIs. Especially because of the Government's dual role, PM must be done with integrity and shielded from undue political influence, in the way that National Statistics are shielded. It is in everyone's interest that Ministers, Parliament, the professions, practitioners and the wider public can have confidence in the PM process, and find the conclusions from it convincing. Before introducing PM in any public service, a PM protocol should be written. This is an orderly record not only of decisions made but also of the reasoning or calculations that led to those decisions. A PM protocol should cover objectives, design considerations and the definition of PIs, sampling versus complete enumeration, the information to be collected about context, the likely perverse behaviours or side-effects that might be induced as a reaction to the monitoring process, and also the practicalities of implementation. Procedures for data collection, analysis, presentation of uncertainty and adjustment for context, together with dissemination rules, should be explicitly defined and reflect good statistical practice. Because of their usually tentative nature, PIs should be seen as ‘screening devices’ and not overinterpreted. If quantitative performance targets are to be set, they need to have a sound basis, take account of prior (and emerging) knowledge about key sources of variation, and be integral to the PM design. Aspirational targets have a distinctive role, but one which is largely irrelevant in the design of a PM procedure; motivational targets which are not rationally based may demoralize and distort. Anticipated and actual side-effects of PM, including on individuals’ behaviour and priorities, may need to be monitored as an intrinsic part of the PM process. Independent scrutiny of PM schemes for the public services should be set up and must report publicly. The extent and nature of this scrutiny should be related to the assessed drawbacks and benefits, reflect ethical concerns, and conform with good statistical practice. Research is needed into the merits of different strategies for identifying institutions or individuals in the public release of PM data, into how new PM schemes should be evaluated, and into efficient designs for evaluating a series of new policies which are monitored by PIs. The Royal Statistical Society considers that attempts to educate the wider public, as well as policy makers, about the issues surrounding the use of PIs are very important. High priority should be given to sponsoring well-informed public debate, and to disseminating good practices by implementing them across Government.

Evaluating the effect of education on earnings: models, methods and results from the National Child Development Survey

Abstract: Regression, matching, control function and instrumental variables methods for recovering the effect of education on individual earnings are reviewed for single treatments and sequential multiple treatments with and without heterogeneous returns. The sensitivity of the estimates once applied to a common data set is then explored. We show the importance of correcting for detailed test score and family background differences and of allowing for (observable) heterogeneity in returns. We find an average return of 27% for those completing higher education versus anything less. Compared with stopping at 16 years of age without qualifications, we find an average return to O-levels of 18%, to A-levels of 24% and to higher education of 48%.

Biostatistics: a Methodology for the Health Sciences

Abstract: This versatile reference provides a wide-ranging look at basic and advanced biostatistical concepts and methods in a format calibrated to individual interests and levels of proficiency. Written with an eye toward the use of computer applications, the book examines the design of medical studies, descriptive statistics, and introductory ideas of probability theory and statistical inference; explores more advanced statistical methods; and illustrates important current uses of biostatistics.

Exploring the economic and social determinants of psychological well‐being and perceived social support in England

Abstract: Summary. A fundamental focus of Government concern is to enhance well-being. Recently, policy makers in the UK and elsewhere have recognized the importance of the community and society to the well-being of the nation as a whole. We explore the extent to which economic and social factors influence the psychological well-being of individuals and their perceptions of the social support that they receive, using Health Survey for England data. We employ a random-effects ordered probit modelling approach and find that unobserved intrahousehold characteristics help to explain the variation in our dependent variables, particularly for co-resident females. Our results indicate that individuals with acute and chronic physical illness, who are female, unemployed or inactive in the labour market and who live in poor households or areas of multiple deprivation report lower levels of psychological well-being. Reduced perceptions of social support are associated with being male, single or post marriage, from an ethnic minority, having low educational attainment and living in a poor household, but are not statistically related to area deprivation measures. These findings may help to inform the contemporary policy debate surrounding the promotion of individual well-being and community, through the alleviation of social exclusion.

Releasing multiply imputed, synthetic public use microdata: an illustration and empirical study

Abstract: Summary. The paper presents an illustration and empirical study of releasing multiply imputed, fully synthetic public use microdata. Simulations based on data from the US Current Population Survey are used to evaluate the potential validity of inferences based on fully synthetic data for a variety of descriptive and analytic estimands, to assess the degree of protection of confidentiality that is afforded by fully synthetic data and to illustrate the specification of synthetic data imputation models. Benefits and limitations of releasing fully synthetic data sets are discussed.

Robust, accurate confidence intervals with a weak instrument: quarter of birth and education

Abstract: Summary. An instrument or instrumental variable manipulates a treatment and affects the outcome only indirectly through its manipulation of the treatment. For instance, encouragement to exercise might increase cardiovascular fitness, but only indirectly to the extent that it increases exercise. If instrument levels are randomly assigned to individuals, then the instrument may permit consistent estimation of the effects caused by the treatment, even though the treatment assignment itself is far from random. For instance, one can conduct a randomized experiment assigning some subjects to ‘encouragement to exercise’ and others to ‘no encouragement’ but, for reasons of habit or taste, some subjects will not exercise when encouraged and others will exercise without encouragement; none-the-less, such an instrument aids in estimating the effect of exercise. Instruments that are weak, i.e. instruments that have only a slight effect on the treatment, present inferential problems.We evaluate a recent proposal for permutation inference with an instrumental variable in four ways: using Angrist and Krueger’s data on the effects of education on earnings using quarter of birth as an instrument, following Bound, Jaeger and Baker in using simulated independent observations in place of the instrument in Angrist and Krueger’s data, using entirely simulated data in which correct answers are known and finally using statistical theory to show that only permutation inferences maintain correct coverage rates. The permutation inferences perform well in both easy and hard cases, with weak instruments, as well as with long-tailed responses.

Measuring the efficiency of public services: the limits of analysis

Abstract: Summary. Policy makers are increasingly seeking to develop overall measures of the efficiency of public service organizations. For that, the use of 'off-the-shelf' statistical tools such as data envelopment analysis and stochastic frontier analysis have been advocated as tools to measure organizational efficiency. The analytical sophistication of such methods has reached an advanced stage of development. We discuss the context within which such models are deployed, their underlying assumptions and their usefulness for a regulator of public services. Four specific model building issues are discussed: the weights that are attached to public service outputs; the specification of the statistical model; the treatment of environmental influences on performance; the treatment of dynamic effects. The paper concludes with recommendations for policy makers and researchers on the development and use of efficiency measurement techniques.

Top incomes in the UK over the 20th century

Abstract: Summary. Recent changes in the distribution of income need to be placed in historical context. The paper provides new evidence about the evolution of top incomes in the UK over the 20th century. Making use of published tabulations of the income tax statistics, and of microdata for recent years, we construct estimates of the shares of top income groups, giving for the first time an annual time series for gross incomes that spans more than 90 years. The paper pays particular attention to the problems of data construction and of the interpretation of tax-based evidence. The resulting statistics have evident limitations but throw light on periods, such as that between the First and Second World Wars, for which there is little other empirical material. The results bring out clearly how the major equalization of the first three-quarters of the century in the UK has been reversed, taking the shares of the top income groups back to levels of inequality found 50 years ago. A similar U-shaped pattern is found for the USA, but the post-war experience of France is different from that in the UK.

Survey response and survey characteristics: microlevel evidence from the European Community Household Panel

Abstract: This paper presents micro-level evidence on the role of the socio-demographic characteristics of the population and the characteristics of the data collection process as predictors of survey response. Our evidence is based on the public use files of the European Community Household Panel (ECHP), a longitudinal household survey covering the countries of the European Union, whose attractive feature is the high level of comparability across countries and over time. We model the response process as the outcome of two sequential events: (i) contact between the interviewer and an eligible interviewee, and (ii) cooperation of the interviewee. Our model allows for dependence between the ease of contact and the propensity to cooperate, taking into account the censoring problem caused by the fact that we observe whether a person is a respondent only if she has been contacted.

Age at first marriage in Malawi: a Bayesian multilevel analysis using a discrete time‐to‐event model

Abstract: Summary. The paper presents a hierarchical discrete time survival model for the analysis of the 2000 Malawi Demographic and Health Survey data to assess the determinants of transition to marriage among women in Malawi. The model explicitly accounts for the unobserved heterogeneity by using family and community random effects with cross-level correlation structure. A nonparametric technique is used to model the base-line discrete hazard dynamically. Parameters of the model are computed by using a Markov chain Monte Carlo algorithm. The results show that rising age at marriage is a combination of birth cohort and education effects, depends considerably on the family and to some extent on the community in which a woman resides and the correlation between family and community random effects is negative. These results confirm a downward trend in teenage marriage and that raising women's education levels in subSaharan Africa has the beneficial effect of increasing age at marriage, and by implication reducing total fertility rates. The negative correlation between family and community random effects has policy implications in that targeting communities with an intervention to increase age at first marriage may not necessarily yield reduced fertility levels in individual families. A campaign that is geared towards individual families would achieve the desired goals. Overall, the findings point to the need for the Government in Malawi to enact public policies which are geared at vastly improving women's education at higher levels. The variation in marriage rates over families poses problems in delivering the policy, since particular policies must be devised for specific groups of families to accomplish the required social and health objectives.

A continuous time Markov model for the length of stay of elderly people in institutional long-term care

Abstract: The paper develops a Markov model in continuous time for the length of stay of elderly people moving within and between residential home care and nursing home care. A procedure to determine the structure of the model and to estimate parameters by maximum likelihood is presented. The modelling approach was applied to 4 years' placement data from the social services department of a London borough. The results in this London borough suggest that, for residential home care, a single-exponential distribution with mean 923 days is adequate to provide a good description of the pattern of the length of stay, whereas, for nursing home care, a mixed exponential distribution with means 59 days (short stay) and 784 days (long stay) is required, and that 64% of admissions to nursing home care will become long-stay residents. The implications of these findings and the advantages of the proposed modelling approach in the general context of long-term care are discussed.

Ecological inference for 2 × 2 tables (with discussion)

Abstract: Summary. A fundamental problem in many disciplines, including political science, sociology and epidemiology, is the examination of the association between two binary variables across a series of 2 × 2 tables, when only the margins are observed, and one of the margins is fixed. Two unobserved fractions are of interest, with only a single response per table, and it is this non-identifiability that is the inherent difficulty lying at the heart of ecological inference. Many methods have been suggested for ecological inference, often without a probabilistic model; we clarify the form of the sampling distribution and critique previous approaches within a formal statistical framework, thus allowing clarification and examination of the assumptions that are required under all approaches. A particularly difficult problem is choosing between models with and without contextual effects. Various Bayesian hierarchical modelling approaches are proposed to allow the formal inclusion of supplementary data, and/or prior information, without which ecological inference is unreliable. Careful choice of the prior within such models is required, however, since there may be considerable sensitivity to this choice, even when the model assumed is correct and there are no contextual effects. This sensitivity is shown to be a function of the number of areas and the distribution of the proportions in the fixed margin across areas. By explicitly providing a likelihood for each table, the combination of individual level survey data and aggregate level data is straightforward and we illustrate that survey data can be highly informative, particularly if these data are from a survey of the minority population within each area. This strategy is related to designs that are used in survey sampling and in epidemiology. An approximation to the suggested likelihood is discussed, and various computational approaches are described. Some extensions are outlined including the consideration of multiway tables, spatial dependence and area-specific (contextual) variables. Voter registration–race data from 64 counties in the US state of Louisiana are used to illustrate the methods.

Breaking the boundaries: geographical approaches to integrating 200 years of the census

Abstract: The census and similar sources of data have been published for two centuries so the information that they contain should provide an unparalleled insight into the changing population of Britain over this time period. To date, however, the seemingly trivial problem of changes in boundaries has seriously hampered the use of these sources as they make it impossible to create long run time series of spatially detailed data. The paper reviews methodologies that attempt to resolve this problem by using geographical information systems and areal interpolation to allow the reallocation of data from one set of administrative units onto another. This makes it possible to examine change over time for a standard geography and thus it becomes possible to unlock the spatial detail and the temporal depth that are held in the census and in related sources. © 2005 Royal Statistical Society.

Editorial: Model selection and efficiency—is ‘Which model …?’ the right question?

Abstract: Statistics as a science and profession has been transformed over the last few decades by fine-tuning its orientation to serve other scientific fields, encouraged by the revolution in computing technology. As a result, it encompasses a vast variety of activities, provides a wide range of careers and is universally accepted as indispensable to the modern information society. These positive aspects go hand in hand with profound weaknesses. We cannot agree on an authoritative definition of our subject, on a short list of its fundamental principles (those of probability theory are insufficient) or on what amounts to good practice in particular settings, and how to promote it. Model selection, with the associated uncertainty, is an example of practice that is becoming increasingly problematic as powerful computers and convenient software enable us to explore data in ever greater detail. We can inspect how several alternative models fit the studied data set, and settle on one of them. Such a 'final' model and its maximum likelihood (ML) fit (estimates and standard errors, or information equivalent to them) is the centre-piece of the results section of many a report or manuscript, accompanied by model checking and claims of (approximate) unbiasedness and asymptotic efficiency, after confirming that the requisite regularity conditions have been satisfied. Despite being regarded as respectable, this approach is flawed because it ignores the consequences of model uncertainty. Since the lucid discussions by Draper (1995) and Chatfield (1995), neither research nor practice has paid much attention to this issue. To Bayesians, the topic might be broached constructively by paraphrasing de Finnetti (1974) ('Every probability is conditional') as 'Every posterior distribution is conditional'. We usually study the properties of estimators conditionally on the selected model, ruling out the possibility that the selected model might not be valid. After all, the model selection is a random (data-dependent) process, but the (unknown) 'good' model is fixed, being a property of the studied phenomenon and oblivious to our study design and data collection process. Model selection forces us to make a decision without considering the consequences of the errors that may have been made in the process. We end up putting all our inferential eggs in one unevenly woven basket. Depending on the purpose, an error of one kind may be innocuous or disastrous relative to an error of another kind. The conditional probabilities of these two kinds of error, controlled in hypothesis testing, are often …

The case for small area microdata

Abstract: Census data are available in aggregate form for local areas and, through the samples of anonymized records (SARs), as samples of microdata for households and individuals. In 1991 there were two SAR files: a household file and an individual file. These have a high degree of detail on the census variables but little geographical detail, a situation that will be exacerbated for the 2001 SAR owing to the loss of district level geography on the individual SAR. The paper puts forward the case for an additional sample of microdata, also drawn from the census, that has much greater geographical detail. Small area microdata (SAM) are individual level records with local area identifiers and, to maintain confidentiality, reduced detail on the census variables. Population data from seven local authorities, including rural and urban areas, are used to define prototype samples of SAM. The rationale for SAM is given, with examples that demonstrate the role of local area information in the analysis of census data. Since there is a trade-off between the extent of local detail and the extent of detail on variables that can be made available, the confidentiality risk of SAM is assessed empirically. An indicative specification of the SAM is given, having taken into account the results of the confidentiality analysis.