We present a new technique called “t-SNE” that visualizes high-dimensional data by giving each datapoint a location in a two or three-dimensional map. The technique is a variation of Stochastic Neighbor Embedding (Hinton and Roweis, 2002) that is much easier to optimize, and produces significantly better visualizations by reducing the tendency to crowd points together in the center of the map. t-SNE is better than existing techniques at creating a single map that reveals structure at many different scales. This is particularly important for high-dimensional data that lie on several different, but related, low-dimensional manifolds, such as images of objects from multiple classes seen from multiple viewpoints. For visualizing the structure of very large datasets, we show how t-SNE can use random walks on neighborhood graphs to allow the implicit structure of all of the data to influence the way in which a subset of the data is displayed. We illustrate the performance of t-SNE on a wide variety of datasets and compare it with many other non-parametric visualization techniques, including Sammon mapping, Isomap, and Locally Linear Embedding. The visualizations produced by t-SNE are significantly better than those produced by the other techniques on almost all of the datasets.

/pdf/visualizing-data-using-t-sne-2w1n304iq0.pdf

Visualizing Data using t-SNE

The analysis of censored failure times is considered. It is assumed that on each individual arc available values of one or more explanatory variables. The hazard function (age-specific failure rate) is taken to be a function of the explanatory variables and unknown regression coefficients multiplied by an arbitrary and unknown function of time. A conditional likelihood is obtained, leading to inferences about the unknown regression coefficients. Some generalizations are outlined.

https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517-6161.1972.tb00899.x

Regression Models and Life-Tables

Generalized Additive Models.

An essential textbook for any student or researcher in biology needing to design experiments, sample programs or analyse the resulting data The text begins with a revision of estimation and hypothesis testing methods, covering both classical and Bayesian philosophies, before advancing to the analysis of linear and generalized linear models Topics covered include linear and logistic regression, simple and complex ANOVA models (for factorial, nested, block, split-plot and repeated measures and covariance designs), and log-linear models Multivariate techniques, including classification and ordination, are then introduced Special emphasis is placed on checking assumptions, exploratory data analysis and presentation of results The main analyses are illustrated with many examples from published papers and there is an extensive reference list to both the statistical and biological literature The book is supported by a website that provides all data sets, questions for each chapter and links to software

/pdf/experimental-design-and-data-analysis-for-biologists-8ybva8d53x.pdf

Experimental Design and Data Analysis for Biologists

Wireless Communications

Thus far, genome-wide association studies (GWAS) have been disappointing in the inability of investigators to use the results of identified, statistically significant variants in complex diseases to make predictions useful for personalized medicine. Why are significant variables not leading to good prediction of outcomes? We point out that this problem is prevalent in simple as well as complex data, in the sciences as well as the social sciences. We offer a brief explanation and some statistical insights on why higher significance cannot automatically imply stronger predictivity and illustrate through simulations and a real breast cancer example. We also demonstrate that highly predictive variables do not necessarily appear as highly significant, thus evading the researcher using significance-based methods. We point out that what makes variables good for prediction versus significance depends on different properties of the underlying distributions. If prediction is the goal, we must lay aside significance as the only selection standard. We suggest that progress in prediction requires efforts toward a new research agenda of searching for a novel criterion to retrieve highly predictive variables rather than highly significant variables. We offer an alternative approach that was not designed for significance, the partition retention method, which was very effective predicting on a long-studied breast cancer data set, by reducing the classification error rate from 30% to 8%.

https://www.pnas.org/content/pnas/112/45/13892.full.pdf

Why significant variables aren’t automatically good predictors

/pdf/sequential-tests-for-the-mean-of-a-normal-distribution-3f2hytwzc5.pdf

Sequential Tests for the Mean of a Normal Distribution

A technique is developed to obtain optimal accelerated life designs for estimating the parameters describing the mean lifetime of a device under a standard environment. Five examples involving variations of models and experimental design are studied. An exponential distribution of lifetimes is assumed in each of the five problems. The general approach, applyin g a method of Elfving, is applicable to a large variety of models.

Optimal Accelerated Life Designs for Estimation

: Imagine a spaceship travelling towards a certain planet with predetermined speed, in a direction which will bring it close to the target after a known period of time. Observations on the position of the target, relative to the present course, are made continuously and lead to a gradually improving prediction of the eventual miss distance. On the other hand, the fuel available in the spaceship for making minor changes in the direction of motion, is gradually losing its effectiveness. This is because the final change of position caused by a small velocity imposed perpendicular to the present motion, is roughly proportional to the remaining time. Thus we have a control problem which is essentially one of compromise between the extremes of using the fuel early and perhaps in the wrongway, because of poor information; or waiting too long for more precise information, so that the fuel becomes ineffective.

/pdf/sequential-decisions-in-the-control-of-a-spaceship-5c9uzoyh3w.pdf

Herman Chernoff

Papers

Why significant variables aren’t automatically good predictors

Sequential Tests for the Mean of a Normal Distribution

Optimal Accelerated Life Designs for Estimation

Sequential decisions in the control of a spaceship

Asymptotic distribution of the likelihood ratio test that a mixture of two binomials is a single binomial