The increasing volume of data in modern business and science calls for more complex and sophisticated tools. Although advances in data mining technology have made extensive data collection much easier, it's still always evolving and there is a constant need for new techniques and tools that can help us transform this data into useful information and knowledge. Since the previous edition's publication, great advances have been made in the field of data mining. Not only does the third of edition of Data Mining: Concepts and Techniques continue the tradition of equipping you with an understanding and application of the theory and practice of discovering patterns hidden in large data sets, it also focuses on new, important topics in the field: data warehouses and data cube technology, mining stream, mining social networks, and mining spatial, multimedia and other complex data. Each chapter is a stand-alone guide to a critical topic, presenting proven algorithms and sound implementations ready to be used directly or with strategic modification against live data. This is the resource you need if you want to apply today's most powerful data mining techniques to meet real business challenges. * Presents dozens of algorithms and implementation examples, all in pseudo-code and suitable for use in real-world, large-scale data mining projects. * Addresses advanced topics such as mining object-relational databases, spatial databases, multimedia databases, time-series databases, text databases, the World Wide Web, and applications in several fields. *Provides a comprehensive, practical look at the concepts and techniques you need to get the most out of real business data

/pdf/data-mining-concepts-and-techniques-4dtvdfkvmi.pdf

Data Mining: Concepts and Techniques

The previously developed particle mesh Ewald method is reformulated in terms of efficient B‐spline interpolation of the structure factors This reformulation allows a natural extension of the method to potentials of the form 1/rp with p≥1 Furthermore, efficient calculation of the virial tensor follows Use of B‐splines in place of Lagrange interpolation leads to analytic gradients as well as a significant improvement in the accuracy We demonstrate that arbitrary accuracy can be achieved, independent of system size N, at a cost that scales as N log(N) For biomolecular systems with many thousands of atoms this method permits the use of Ewald summation at a computational cost comparable to that of a simple truncation method of 10 A or less

A smooth particle mesh Ewald method

Mediation is said to occur when a causal effect of some variable X on an outcome Y is explained by some intervening variable M. The authors recommend that with small to moderate samples, bootstrap methods (B. Efron & R. Tibshirani, 1993) be used to assess mediation. Bootstrap tests are powerful because they detect that the sampling distribution of the mediated effect is skewed away from 0. They argue that R. M. Baron and D. A. Kenny's (1986) recommendation of first testing the X --> Y association for statistical significance should not be a requirement when there is a priori belief that the effect size is small or suppression is a possibility. Empirical examples and computer setups for bootstrap analyses are provided.

/pdf/mediation-in-experimental-and-nonexperimental-studies-new-36pi5vkg4y.pdf

Mediation in experimental and nonexperimental studies: New procedures and recommendations.

SUMMARY Multivariable regression models are powerful tools that are used frequently in studies of clinical outcomes. These models can use a mixture of categorical and continuous variables and can handle partially observed (censored) responses. However, uncritical application of modelling techniques can result in models that poorly fit the dataset at hand, or, even more likely, inaccurately predict outcomes on new subjects. One must know how to measure qualities of a model's fit in order to avoid poorly fitted or overfitted models. Measurement of predictive accuracy can be difficult for survival time data in the presence of censoring. We discuss an easily interpretable index of predictive discrimination as well as methods for assessing calibration of predicted survival probabilities. Both types of predictive accuracy should be unbiasedly validated using bootstrapping or cross-validation, before using predictions in a new data series. We discuss some of the hazards of poorly fitted and overfitted regression models and present one modelling strategy that avoids many of the problems discussed. The methods described are applicable to all regression models, but are particularly needed for binary, ordinal, and time-to-event outcomes. Methods are illustrated with a survival analysis in prostate cancer using Cox regression. Accurate estimation of patient prognosis is important for many reasons. First, prognostic estimates can be used to inform the patient about likely outcomes of her disease. Second, the physician can use estimates of prognosis as a guide for ordering additional tests and selecting appropriate therapies. Third, prognostic assessments are useful in the evaluation of technologies; prognostic estimates derived both with and without using the results of a given test can be compared to measure the incremental prognostic information provided by that test over what is provided by prior information.' Fourth, a researcher may want to estimate the effect of a single factor (for example, treatment given) on prognosis in an observational study in which many uncontrolled confounding factors are also measured. Here the simultaneous effects of the uncontrolled variables must be controlled (held constant mathematically if using a regression model) so that the effect of the factor of interest can be more purely estimated. An analysis of how variables (especially continuous ones) affect the patient outcomes of interest is necessary to

/pdf/tutorial-in-biostatistics-multivariable-prognostic-models-4auy6032o6.pdf

Tutorial in biostatistics multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors

In this paper, the authors show that PLS path modeling can be used to assess a hierarchical construct model. They provide guidelines outlining four key steps to construct a hierarchical construct model using PLS path modeling. This approach is illustrated empirically using a reflective, fourth-order latent variable model of online experiential value in the context of online book and CD retailing. Moreover, the guidelines for the use of PLS path modeling to estimate parameters in a hierarchical construct model are extended beyond the scope of the empirical illustration. The findings of the empirical illustration are used to discuss the use of covariance-based SEM versus PLS path modeling. The authors conclude with the limitations of their study and suggestions for future research.

/pdf/using-pls-path-modeling-for-assessing-hierarchial-construct-1nuf1tphs9.pdf

Using PLS path modeling for assessing hierarchial construct models: guidelines and impirical illustration

Statistics is the science of learning from experience, especially experience that arrives a little bit at a time. The earliest information
science was statistics, originating in about 1650. This century has
seen statistical techniques become the analytic methods of choice
in biomedical science, psychology, education, economics, communications theory, sociology, genetic studies, epidemiology, and other
areas. Recently, traditional sciences like geology, physics, and astronomy have begun to make increasing use of statistical methods
as they focus on areas that demand informational efficiency, such as
the study of rare and exotic particles or extremely distant galaxies.
Most people are not natural-born statisticians. Left to our own
devices we are not very good at picking out patterns from a sea
of noisy data. To put it another way, we are all too good at picking out non-existent patterns that happen to suit our purposes.
Statistical theory attacks the problem from both ends. It provides
optimal methods for finding a real signal in a noisy background,
and also provides strict checks against the overinterpretation of
random patterns.

An Introduction to the Bootstrap

The generalized Pareto distribution (GPD) is a two-parameter family of distributions that can be used to model exceedances over a threshold. Maximum likelihood estimators of the parameters are preferred, since they are asymptotically normal and asymptotically efficient in many cases. Numerical methods are required for maximizing the log-likelihood, however. This article investigates the properties of a reduction of the two-dimensional numerical search for the zeros of the log-likelihood gradient vector to a one-dimensional numerical search. An algorithm for computing the GPD maximum likelihood estimates based on this dimension reduction and properties are given.

Computing Maximum Likelihood Estimates for the Generalized Pareto Distribution

We show how agency problems between lenders (principals) and third–party originators (TPO; agents) imply that TPO–originated loans are more likely to default than similar retail–originated loans. The nature of the agency problem is that TPOs are compensated for writing loans, but are not completely held accountable for the subsequent performance of those loans. Using a hazard model with jointly estimated competing risks and unobserved heterogeneity, we find empirical support for the TPO/default prediction using individual fixed–rate subprime loans with first liens secured by residential real estate originated between January 1, 1996, and December 31, 1998. We find that apparently equal loans (similar ability to pay, option incentives and term) can have unequal default probabilities. We also find that, initially, the agency–cost risk was not priced. At first, the market did not recognize the higher channel risk, since TPO and retail loans received similar interest rates even though the TPO loans were more likely to default. We also show that this inefficiency was short–lived. As the difference in default rates became apparent, interest rates on TPO loans rose about 50 basis points above otherwise similar retail loans.

Some Loans Are More Equal than Others: Third–Party Originations and Defaults in the Subprime Mortgage Industry

A Markov chain is a natural probability model for accounts receivable. For example, accounts that are ‘current’ this month have a probability of moving next month into ‘current’, ‘delinquent’ or ‘paid-off’ states. If the transition matrix of the Markov chain were known, forecasts could be formed for future months for each state. This paper applies a Markov chain model to subprime loans that appear neither homogeneous nor stationary. Innovative estimation methods for the transition matrix are proposed. Bayes and empirical Bayes estimators are derived where the population is divided into segments or subpopulations whose transition matrices differ in some, but not all entries. Loan-level models for key transition matrix entries can be constructed where loan-level covariates capture the non-stationarity of the transition matrix. Prediction is illustrated on a $7 billion portfolio of subprime fixed first mortgages and the forecasts show good agreement with actual balances in the delinquency states. Copyright © 2010 John Wiley & Sons, Ltd.

Markov chain models for delinquency: Transition matrix estimation and forecasting

A control chart is proposed which monitors the conformance of a sample to an in-control distribution using the quantile or inverse cumulative distribution function. This method permits detecting changes in the distributional shape which may be undetecte..

Scott D. Grimshaw

Papers

An Introduction to the Bootstrap

Computing Maximum Likelihood Estimates for the Generalized Pareto Distribution

Some Loans Are More Equal than Others: Third–Party Originations and Defaults in the Subprime Mortgage Industry

Markov chain models for delinquency: Transition matrix estimation and forecasting

Control Charts for Quantile Function Values