https://www.onlinecjc.ca/article/S0828-282X(14)00005-1/pdf

Next-Generation Airbags and the Possibility of Negative Outcomes Due to Thoracic Injury

Predictive accuracy claims should give explicit descriptions of the steps followed, with access to the code used. This allows referees and readers to check for common traps, and to repeat the same steps on other data. Feature selection and/or model selection and/or tuning must be independent of the test data. For use of cross-validation, such steps must be repeated at each fold. Even then, such accuracy assessments have the limitation that the target population, to which results will be applied, is commonly different from the source population. Commonly, it is shifted forward in time, and it may differ in other respects also.

A consequence of source/target differences is that highly sophisticated modeling may be pointless or even counter-productive. At best, model effects in the target population may be broadly similar. Investigation of the pattern of changes over time is required. Such studies are unusual in the data mining literature, in part because relevant data have not been available.

Several recent investigations are noted that shed interesting light on the comparison between observational and experimental studies, with particular relevance when there is an interest in giving parameter estimates a causal interpretation.

Data mining activity would benefit from wider co-operation in the development and deployment of computing tools, and from better integration of those tools into the publication process.

/pdf/data-mining-methodological-weaknesses-and-suggested-fixes-43vba69w84.pdf

Data mining methodological weaknesses and suggested fixes

If a front-seat occupant wishes to ask the question, “If I get in an accident, am I less likely or more likely to die, if I have an airbag?” the proper way to answer this question is with the CDS dataset. With the FARS dataset, the question one can answer is, “If I get in a highway accident in which there is at least one fatality, am I less likely or more likely to die, if I have an airbag?” It seems paradoxical that these two questions can have different answers, but they do. The CDS dataset can show us that in low-speed collisions, having an airbag increases the probability of death, and this is especially true for unseatbelted occupants where the main collision is from the side. This fact cannot be seen using the FARS dataset, because the information about low-speed crashes in which there was not a fatality is missing. The increase in risk to occupants in low-speed crashes, due to airbags, cannot be demonstrated. Therefore, the CDS dataset is the proper tool to assess risk of death in an accident. The distinction is not at all obvious at the outset, and no blame should be attached to those who chose to use FARS for the original analyses. The airbag risk analysis is a great example of the subtleties and challenges of quantitative reasoning, but it is clearly demonstrated that the CDS analyses better reflect the truth about airbags. Not only can we reproduce the results from the analyses with FARS, but there are convincing explanations for the disparate results. This country’s commitment to airbags as a safety device needs to be reexamined immediately.

MC Meyer

Papers

Reply to discussion of "Who wants airbags?"