Showing papers in "Chance in 2005"

Why Most Published Research Findings Are False

Abstract: Summary There is increasing concern that most current published research fi ndings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientifi c fi eld. In this framework, a research fi nding is less likely to be true when the studies conducted in a fi eld are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater fl exibility in designs, defi nitions, outcomes, and analytical modes; when there is greater fi nancial and other interest and prejudice; and when more teams are involved in a scientifi c fi eld in chase of statistical signifi cance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientifi c fi elds, claimed research fi ndings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research. It can be proven that most claimed research fi ndings are false.

Was It Ethnic and Social-Class Bias or Statistical Artifact? Logical and Empirical Evidence against Freedle's Method for Reestimating SAT Scores

Abstract: (2005). Was It Ethnic and Social-Class Bias or Statistical Artifact? Logical and Empirical Evidence against Freedle's Method for Reestimating SAT Scores. CHANCE: Vol. 18, Who Wants Airbags?, pp. 17-24.

Analysis of Reserve and Regular Bottlings: Why Pay for a Difference Only the Critics Claim to Notice?

Abstract: I report my tests of the hypothesis that wine consumers cannot distinguish the difference between regular and reserve bottlings of otherwise-similar wines. My results suggest that only about one purchaser in five should even consider buying the reserve bottling of a wine, rather than the regular. But since I cannot tell you how to distinguish that one person out of five, we can all aspire to drinking the reserve. I have more than 800 observations of wine drinkers’ who engaged in the following experiment. The drinker faces 3 glasses of wine, two of which contain identical wines [either the regular or the reserve] and the third contains a different wine [the other one]. I record whether the drinker can distinguish wines whether he can tell the singleton from the doubleton and, if the drinker can distinguish, which wine he prefers. I find that just over 40 per cent of the drinkers distinguish correctly, whereas one-third could if the process were random. Of those 40 percent who can distinguish, 52 percent prefer the more expensive, reserve wine, where as half would if the process were random. For this data set, 52 percent does not differ significantly from the expectedif-random half. I have recorded the sex of the testers and I can find that men can distinguish the wines better than random, but women cannot. The differences are so small, even though significant, however, that the Exact F test detects no significant difference between the ability of men and women in these tests. The results span tests of wines from Bordeaux, Burgundy, the Rhone, Spain, New Zealand, Italy, California [both red and white], Oregon, and Australia; the tests include still and sparkling wines. 1 Also, V. Duane Rath Professor of Accounting, Graduate School of Business, University of Chicago, 1101 East 58 Street, Chicago IL 60637; 773.702.7261; fa x 206.202.2114; roman.weil@gsb.uchicago.edu. I prepared this paper for the 9 Oenometrics Meeting of the Vineyard Data Quantification Society, Budapest, May 22-24, 2003, whose attendees provide my final data set. Some past attendees asked why is an accountant doing such work. I respond as follows. First, note that accounting is an intellectual discipline even though you probably think of it as bookkeeping and tax reporting. Accounting records in aggregating numbers information about complex transactions. Then, it attempts to enable users of that information to deduce from the numbers the underlying reality and how to use the data in decision making. Similarly, wine ratings record in aggregating numbers useful information about complex sensory experience. This paper attempts to help the user decide how to use the reported data in making decisions. Shannon Harwick assisted in the compilation of the data for this research and in the writing of this paper. Neither he nor I can distinguish better than random. M.L. Marais helped us with choosing suitable tests for the sex test. He can.

Who wants airbags

Abstract: Using the National Automotive Sampling System (NASS) Crashworthiness Data System (CDS) we have looked at the effectiveness of airbags on the probability of dying in an accident, controlling for other factors. We found that airbags are actually associated with increased probability of death in accidents. The situations in which airbags are most dangerous are low-speed collisions when the occupants are not wearing seatbelts. How can we reconcile our findings with previous studies that found airbags to be saving lives? For example, the study by Crandall, Olson, and Skiar (2001) is similar to ours, in that a logistic regression model is used to estimate odds ratios and the same effects are controlled for in the model. They looked at head-on collisions in passenger cars, and found the odds ratio for deployment of the airbags is 0.71, with 95 percent CI (.58, .87). Their study used a different NHTSA database called the Fatality Analysis Reporting System (FARS). When we look at the random sample of all accidents (CDS), we get that airbags are associated with increased risk of death, and this increase is due mostly to more deaths with airbags in low-speed crashes and no seatbelts. However, if we limit the dataset to include only collisions in which a fatality occurred, we get a significantly reduced risk of death due to airbags. This is analogous to doing a comparison with the same numerators but different denominators—it is not unlikely that we will get different results. Here is a more dramatic analogy: If you look at people who have some types of cancer, you will see that those who get radiation treatment have a better chance of surviving than those who don’t. However, radiation is inherently dangerous, and could actually cause cancer. If you give everyone radiation treatments whether they have cancer or not, you will probably find an increased risk of death in the general population. Making everyone have airbags and then verifying the effectiveness using only fatal crashes is like making everyone have radiation and then estimating lives saved by looking only at people who have cancer. Overall, there will be more deaths if everyone is given radiation, but in the cancer subset, radiation will be effective. Using the CDS dataset for the analysis, we find that airbags are associated with significantly more deaths, rather than fewer. Using a subset of the CDS data, we can mimic previous analysis and reproduce the results. Because of this, we are confident that our analyses better reflect the actual effectiveness of airbags in the general population. The evidence shows that airbags do more harm than good.

Average Speed Bumps: Four Perspectives on Averaging Speeds

Abstract: highway, where each car travels at a constant speed Assume that the distribution of the speeds is the same throughout the length of the highway You adjust your speed so that during a given time unit you overtake the same number of cars as the number of cars that overtake you Does this mean that your speed is the median of the speeds of the cars on the highway? Surprisingly, the answer is no! (Clevenson, Schilling, Watkins, & Watkins, 2001) Imagine further a radar device at the side of the highway, measuring and recording the speeds of all the cars that pass this point within a fixed time interval Again, contrary to lay expectations, the arithmetic mean of these recordings would generally not reproduce the arithmetic mean of the speeds of all the cars on the highway (Stein & Dattero, 1985) The above examples illustrate that identifying the correct average may have its difficulties (the correct answers for both cases will be detailed later) Average speed, in general, is not all that self-evident a concept The apparently simple question “what is the average speed of the cars that drive on the highway?” is equivocal As students of introductory statistics know, the term average may be interpreted in various ways and hence may assume several different forms One needs to know in what sense the average is supposed to represent a set of observations Four Faces of the Average Speed

Do Fatal Crashes Increase Following a Super Bowl Telecast

Abstract: Can the Super Bowl telecast kill people? Some theories suggest that wild entertainment encourages wild behavior; if so, the telecast might increase fatal crashes. Other theories suggest that exposure to violent images induces satiation and trepidation; if so, the telecast might decrease fatal crashes. Still others assert that broadcasts mirror reality but do not shape reality; if so, the telecast would be predicted to have no effect on than the World Series in baseball and 20 times larger than the Stanley Cup in hockey). The prototypical audience member for the Super Bowl is a young adult in good health, the same type of person who does not seek safety advice yet is overrepresented in fatal crashes. We wondered whether the Super Bowl telecast might thereby provide fresh insights on both motor vehicle crashes and major television broadcasts. M vehicle crashes are a leading cause of death, disability, and destruction. This year, more than 1 million people will die from crashes, according to the World Health Organization, exceeding deaths due to malaria for the first time in history. Saving these lives won’t be accomplished by the human genome project or other miracles of modern medicine because about half of all deaths occur at the scene before an ambulance arrives, leaving no time for heroics. In contrast, almost all the deaths reflect driver error that could be prevented by a small change in behavior. Broadcast television is another major feature of modern society; about 98% of American households own a television and average usage exceeds 50 hours per week. Many have worried about adverse health effects because television shows may lead to a glorification of violence, disregard for the consequences of injury, and imitative behavior in the community. Entertainment also has opportunity costs, since being sedentary contributes to obesity. Moreover, fictional medical dramas often lead to unrealistic beliefs about the effectiveness of medical treatments. The Super Bowl is the most popular regular telecast in the United States, with an estimated viewing audience of 130 million Americans (four times larger Should you avoid driving immediately after the Super Bowl and similar events?

On Solving an Aspect of Unfairness in the Tennis Doubles Scoring System

Abstract: (2005). On Solving an Aspect of Unfairness in the Tennis Doubles Scoring System. CHANCE: Vol. 18, Truth is Slower than Fiction: Francis Galton as an Illustration, pp. 17-19.

Just How Predictable Are the Oscars

Abstract: (2005). Just How Predictable Are the Oscars? CHANCE: Vol. 18, Truth is Slower than Fiction: Francis Galton as an Illustration, pp. 32-39.

Politics, Drugs, and the Olympics: And the Winners Are… Politics and Drugs

Abstract: In 1891, five years before the start of the modern Olympic Games, a Dominican priest named Henri Didon, Prior of Arcueil College in France, spoke to his students at a sports club meeting. Didon taught that an athlete is not a true success until the athlete has tried the utmost to perform faster, higher, and stronger than his or her previous performances. Such was the original meaning of Citius, Altius, Fortius, the Latin translation of “faster, higher, stronger,” which he used to end his speech. To Didon, the athlete’s antagonist was the athlete’s own self. Citius, Altius, Fortius was chosen as the Olympic motto in an organizational meeting of the International Olympic Committee, IOC, in 1894. Over time, Citius, Altius, Fortius has come to mean that the performances of a prototypical Olympic athlete should be faster, higher, and stronger comA careful analysis of Olympic results yields insight into the role of drugs.

Afraid To Discuss Evolution

Abstract: P can try to settle intellectual debates in several ways, chief among them raising their voices higher or countering their opponents with statistical arguments. The second strategy is surely better, but only if the statistics are sound. Unfortunately, when it comes to the current debate about teaching “creation science” and “intelligent design” in the public schools, misleading statistics are front and center. On February 5, 2005, The New York Times published the editorial “Afraid to Discuss Evolution,” which argued there is “ample evidence that even when evolution is theoretically part of the curriculum, it is often ignored or played down in the classroom.” To bolster its position, the Times cited a “1998 doctoral dissertation [,which] found that 24 percent of the biology teachers sampled in Louisiana said that creationism had a scientific foundation and that 17 percent were not sure.” I found these statistics mind-boggling. Could 41% of the biology teachers in Louisiana really reject evolution? The evidence came from a relatively recent dissertation, so perhaps its findings reflected current-day beliefs. Yet I wanted to know more. Who conducted the study? Did the doctoral candidate have a particular perspective on this usually partisan debate and, if so, was this communicated—inadvertently or directly—to the respondents? Was the sample representative? Was the sample size adequate? Was the response rate high? Every newspaper reader should ask questions like these when evaluating scientific evidence, whatever an article’s thesis. So, too, should newspaper writers and editorial boards, including those at The New York Times.

Statistical Sleuthing During Epidemics: Maternal Influenza and Schizophrenia

Abstract: Schizophrenia is among the most debilitating of mental illnesses (NARSAD 2003). Although relatively rare, affecting less than 1% of the population, this form of psychosis is the focus of a great deal of study due to its etiological complexity and its devastating social, economic, and personal effects. The National Institute of Mental Health estimates that schizophrenia is responsible for 2.3 million lost years of healthy life in America (NIMH 2001) and there is a huge economic cost burden associated with the disease [$7 billion dollars for treatment annually in the U.S., by one estimate (Bromet 1995)]. Schizophrenics typically experience frightening hallucinations or bizarre and persistent delusions, sometimes in conjunction with paranoia. Many have disorganized thought and speech patterns and have absent or diminished outward emotional expression. While recent advances in psychopharmacology offer relief of some symptoms, schizophrenia remains an incurable, lifelong illness and most patients remain impaired to a large degree. Is schizophrenia related to birthdate? This article presents data and possible confounding variables.

Chance at the Bar

Abstract: (2005). CHANCE at the Bar. CHANCE: Vol. 18, Missile Detection and Tracking in Satellite Imagery, pp. 34-37.

Reply to discussion of "Who wants airbags?"

Abstract: If a front-seat occupant wishes to ask the question, “If I get in an accident, am I less likely or more likely to die, if I have an airbag?” the proper way to answer this question is with the CDS dataset. With the FARS dataset, the question one can answer is, “If I get in a highway accident in which there is at least one fatality, am I less likely or more likely to die, if I have an airbag?” It seems paradoxical that these two questions can have different answers, but they do. The CDS dataset can show us that in low-speed collisions, having an airbag increases the probability of death, and this is especially true for unseatbelted occupants where the main collision is from the side. This fact cannot be seen using the FARS dataset, because the information about low-speed crashes in which there was not a fatality is missing. The increase in risk to occupants in low-speed crashes, due to airbags, cannot be demonstrated. Therefore, the CDS dataset is the proper tool to assess risk of death in an accident. The distinction is not at all obvious at the outset, and no blame should be attached to those who chose to use FARS for the original analyses. The airbag risk analysis is a great example of the subtleties and challenges of quantitative reasoning, but it is clearly demonstrated that the CDS analyses better reflect the truth about airbags. Not only can we reproduce the results from the analyses with FARS, but there are convincing explanations for the disparate results. This country’s commitment to airbags as a safety device needs to be reexamined immediately.

The Impact of Technology on the Scientific Method

Abstract: (2005). The Impact of Technology on the Scientific Method. CHANCE: Vol. 18, Truth is Slower than Fiction: Francis Galton as an Illustration, pp. 4-8.

The Effectiveness of SAT Coaching on Math SAT Scores

Abstract: (2005). The Effectiveness of SAT Coaching on Math SAT Scores. CHANCE: Vol. 18, Who Wants Airbags?, pp. 25-34.

Risky Business: The Use and Misuse of Statistics in Casino Gaming

Abstract: (2005). Risky Business: The Use and Misuse of Statistics in Casino Gaming. CHANCE: Vol. 18, Who Wants Airbags?, pp. 41-47.

Comment: Anova as a Tool for Structuring and Understanding Hierarchical Models

Abstract: (2005). Comment: Anova as a Tool for Structuring and Understanding Hierarchical Models. CHANCE: Vol. 18, Missile Detection and Tracking in Satellite Imagery, pp. 33-33.

Here'S to Your Health: What's the Big Deal about BMI?

Abstract: (2005). Here'S to Your Health. CHANCE: Vol. 18, Missile Detection and Tracking in Satellite Imagery, pp. 49-52.