Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

General Strategies.- Regression Models.- Classification Models.- Other Considerations.- Appendix.- References.- Indices.

Applied Predictive Modeling

Statistical Analysis with Missing Data

Variable importance measures for random forests have been receiving increased attention as a means of variable selection in many classification tasks in bioinformatics and related scientific fields, for instance to select a subset of genetic markers relevant for the prediction of a certain disease. We show that random forest variable importance measures are a sensible means for variable selection in many applications, but are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories. This is particularly important in genomics and computational biology, where predictors often include variables of different types, for example when predictors include both sequence data and continuous variables such as folding energy, or when amino acid sequence data show different numbers of categories. Simulation studies are presented illustrating that, when random forest variable importance measures are used with data of varying types, the results are misleading because suboptimal predictor variables may be artificially preferred in variable selection. The two mechanisms underlying this deficiency are biased variable selection in the individual classification trees used to build the random forest on one hand, and effects induced by bootstrap sampling with replacement on the other hand. We propose to employ an alternative implementation of random forests, that provides unbiased variable selection in the individual classification trees. When this method is applied using subsampling without replacement, the resulting variable importance measures can be used reliably for variable selection even in situations where the potential predictor variables vary in their scale of measurement or their number of categories. The usage of both random forest algorithms and their variable importance measures in the R system for statistical computing is illustrated and documented thoroughly in an application re-analyzing data from a study on RNA editing. Therefore the suggested method can be applied straightforwardly by scientists in bioinformatics research.

https://link.springer.com/content/pdf/10.1186%2F1471-2105-8-25.pdf

Bias in random forest variable importance measures: Illustrations, sources and a solution

Random forests are becoming increasingly popular in many scientific fields because they can cope with "small n large p" problems, complex interactions and even highly correlated predictor variables. Their variable importance measures have recently been suggested as screening tools for, e.g., gene expression studies. However, these variable importance measures show a bias towards correlated predictor variables. We identify two mechanisms responsible for this finding: (i) A preference for the selection of correlated predictors in the tree building process and (ii) an additional advantage for correlated predictor variables induced by the unconditional permutation scheme that is employed in the computation of the variable importance measure. Based on these considerations we develop a new, conditional permutation scheme for the computation of the variable importance measure. The resulting conditional variable importance reflects the true impact of each predictor variable more reliably than the original marginal approach.

/pdf/conditional-variable-importance-for-random-forests-yw6yxv0jzu.pdf

Conditional variable importance for random forests

Simulation studies present an important statistical tool to investigate the performance, properties and adequacy of statistical models in pre-specified situations. One of the most important statistical models in medical research is the proportional hazards model of Cox. In this paper, techniques to generate survival times for simulation studies regarding Cox proportional hazards models are presented. A general formula describing the relation between the hazard and the corresponding survival time of the Cox model is derived, which is useful in simulation studies. It is shown how the exponential, the Weibull and the Gompertz distribution can be applied to generate appropriate survival times for simulation studies. Additionally, the general relation between hazard and survival time can be used to develop own distributions for special situations and to handle flexibly parameterized proportional hazards models. The use of distributions other than the exponential distribution is indispensable to investigate the characteristics of the Cox proportional hazards model, especially in non-standard situations, where the partial likelihood depends on the baseline hazard. A simulation study investigating the effect of measurement errors in the German Uranium Miners Cohort Study is considered to illustrate the proposed simulation techniques and to emphasize the importance of a careful modelling of the baseline hazard in Cox models.

/pdf/generating-survival-times-to-simulate-cox-proportional-5e3v9se3y4.pdf

Generating survival times to simulate Cox proportional hazards models

Preface Introduction Acknowledgements Outline of this Book and Guide to Readers Contributors 1 Desirability 1.1 Introduction 1.2 Reasoning about and with Sets of Desirable Gambles 1.2.1 Rationality Criteria 1.2.2 Assessments Avoiding Partial or Sure Loss 1.2.3 Coherent Sets of Desirable Gambles 1.2.4 Natural Extension 1.2.5 Desirability Relative to Subspaces with Arbitrary Vector Orderings 1.3 Deriving & Combining Sets of Desirable Gambles 1.3.1 Gamble Space Transformations 1.3.2 Derived Coherent Sets of Desirable Gambles 1.3.3 Conditional Sets of Desirable Gambles 1.3.4 Marginal Sets of Desirable Gambles 1.3.5 Combining Sets of Desirable Gambles 1.4 Partial Preference Orders 1.4.1 Strict Preference 1.4.2 Nonstrict Preference 1.4.3 Nonstrict Preferences Implied by Strict Ones 1.4.4 Strict Preferences Implied by Nonstrict Ones 1.5 Maximally Committal Sets of Strictly Desirable Gambles 1.6 Relationships with Other, Nonequivalent Models 1.6.1 Linear Previsions 1.6.2 Credal Sets 1.6.3 To Lower and Upper Previsions 1.6.4 Simplified Variants of Desirability 1.6.5 From Lower Previsions 1.6.6 Conditional Lower Previsions 1.7 Further Reading 2 Lower Previsions 2.1 Introduction 2.2 Coherent Lower Previsions 2.2.1 Avoiding Sure Loss and Coherence 2.2.2 Linear Previsions 2.2.3 Sets of Desirable Gambles 2.2.4 Natural Extension 2.3 Conditional Lower Previsions 2.3.1 Coherence of a Finite Number of Conditional Lower Previsions 2.3.2 Natural Extension of Conditional Lower Previsions 2.3.3 Coherence of an Unconditional and a Conditional Lower Prevision 2.3.4 Updating with the Regular Extension 2.4 Further Reading 2.4.1 The Work of Williams 2.4.2 The Work of Kuznetsov 2.4.3 The Work of Weichselberger 3 Structural Judgements 3.1 Introduction 3.2 Irrelevance and Independence 3.2.1 Epistemic Irrelevance 3.2.2 Epistemic Independence 3.2.3 Envelopes of Independent Precise Models 3.2.4 Strong Independence 3.2.5 The Formalist Approach to Independence 3.3 Invariance 3.3.1 Weak Invariance 3.3.2 Strong Invariance 3.4 Exchangeability. 3.4.1 Representation Theorem for Finite Sequences 3.4.2 Exchangeable Natural Extension 3.4.3 Exchangeable Sequences 3.5 Further Reading 3.5.1 Independence. 3.5.2 Invariance 3.5.3 Exchangeability 4 Special Cases 4.1 Introduction 4.2 Capacities and n-monotonicity 4.3 2-monotone Capacities 4.4 Probability Intervals on Singletons 4.5 1-monotone Capacities 4.5.1 Constructing 1-monotone Capacities 4.5.2 Simple Support Functions 4.5.3 Further Elements 4.6 Possibility Distributions, p-boxes, Clouds and Related Models. 4.6.1 Possibility Distributions 4.6.2 Fuzzy Intervals 4.6.3 Clouds 4.6.4 p-boxes. 4.7 Neighbourhood Models 4.7.1 Pari-mutuel 4.7.2 Odds-ratio 4.7.3 Linear-vacuous 4.7.4 Relations between Neighbourhood Models 4.8 Summary 5 Other Uncertainty Theories Based on Capacities 5.1 Imprecise Probability = Modal Logic + Probability 5.1.1 Boolean Possibility Theory and Modal Logic 5.1.2 A Unifying Framework for Capacity Based Uncertainty Theories 5.2 From Imprecise Probabilities to Belief Functions and Possibility Theory 5.2.1 Random Disjunctive Sets 5.2.2 Numerical Possibility Theory 5.2.3 Overall Picture 5.3 Discrepancies between Uncertainty Theories 5.3.1 Objectivist vs. Subjectivist Standpoints 5.3.2 Discrepancies in Conditioning 5.3.3 Discrepancies in Notions of Independence 5.3.4 Discrepancies in Fusion Operations 5.4 Further Reading 6 Game-Theoretic Probability 6.1 Introduction 6.2 A Law of Large Numbers 6.3 A General Forecasting Protocol 6.4 The Axiom of Continuity 6.5 Doob s Argument 6.6 Limit Theorems of Probability 6.7 Levy s Zero-One Law. 6.8 The Axiom of Continuity Revisited 6.9 Further Reading 7 Statistical Inference 7.1 Background and Introduction 7.1.1 What is Statistical Inference? 7.1.2 (Parametric) Statistical Models and i.i.d. Samples 7.1.3 Basic Tasks and Procedures of Statistical Inference 7.1.4 Some Methodological Distinctions 7.1.5 Examples: Multinomial and Normal Distribution 7.2 Imprecision in Statistics, some General Sources and Motives 7.2.1 Model and Data Imprecision Sensitivity Analysis and Ontological Views on Imprecision 7.2.2 The Robustness Shock, Sensitivity Analysis 7.2.3 Imprecision as a Modelling Tool to Express the Quality of Partial Knowledge 7.2.4 The Law of Decreasing Credibility 7.2.5 Imprecise Sampling Models: Typical Models and Motives 7.3 Some Basic Concepts of Statistical Models Relying on Imprecise Probabilities 7.3.1 Most Common Classes of Models and Notation 7.3.2 Imprecise Parametric Statistical Models and Corresponding i.i.d. Samples. 7.4 Generalized Bayesian Inference 7.4.1 Some Selected Results from Traditional Bayesian Statistics. 7.4.2 Sets of Precise Prior Distributions, Robust Bayesian Inference and the Generalized Bayes Rule 7.4.3 A Closer Exemplary Look at a Popular Class of Models: The IDM and Other Models Based on Sets of Conjugate Priors in Exponential Families. 7.4.4 Some Further Comments and a Brief Look at Other Models for Generalized Bayesian Inference 7.5 Frequentist Statistics with Imprecise Probabilities 7.5.1 The Non-robustness of Classical Frequentist Methods. 7.5.2 (Frequentist) Hypothesis Testing under Imprecise Probability: Huber-Strassen Theory and Extensions 7.5.3 Towards a Frequentist Estimation Theory under Imprecise Probabilities Some Basic Criteria and First Results 7.5.4 A Brief Outlook on Frequentist Methods 7.6 Nonparametric Predictive Inference (NPI) 7.6.1 Overview 7.6.2 Applications and Challenges 7.7 A Brief Sketch of Some Further Approaches and Aspects 7.8 Data Imprecision, Partial Identification 7.8.1 Data Imprecision 7.8.2 Cautious Data Completion 7.8.3 Partial Identification and Observationally Equivalent Models 7.8.4 A Brief Outlook on Some Further Aspects 7.9 Some General Further Reading 7.10 Some General Challenges 8 Decision Making 8.1 Non-Sequential Decision Problems 8.1.1 Choosing From a Set of Gambles 8.1.2 Choice Functions for Coherent Lower Previsions 8.2 Sequential Decision Problems 8.2.1 Static Sequential Solutions: Normal Form 8.2.2 Dynamic Sequential Solutions: Extensive Form 8.3 Examples and Applications 8.3.1 Ellsberg s Paradox 8.3.2 Robust Bayesian Statistics 9 Probabilistic Graphical Models 9.1 Introduction 9.2 Credal Sets 9.2.1 Definition and Relation with Lower Previsions 9.2.2 Marginalisation and Conditioning 9.2.3 Composition. 9.3 Independence 9.4 Credal Networks 9.4.1 Non-Separately Specified Credal Networks 9.5 Computing with Credal Networks 9.5.1 Credal Networks Updating 9.5.2 Modelling and Updating with Missing Data 9.5.3 Algorithms for Credal Networks Updating 9.5.4 Inference on Credal Networks as a Multilinear Programming Task 9.6 Further Reading 10 Classification 10.1 Introduction 10.2 Naive Bayes 10.3 Naive Credal Classifier (NCC) 10.4 Extensions and Developments of the Naive Credal Classifier 10.4.1 Lazy Naive Credal Classifier 10.4.2 Credal Model Averaging 10.4.3 Profile-likelihood Classifiers 10.4.4 Tree-Augmented Networks (TAN) 10.5 Tree-based Credal Classifiers 10.5.1 Uncertainty Measures on Credal Sets. The Maximum Entropy Function. 10.5.2 Obtaining Conditional Probability Intervals with the Imprecise Dirichlet Model 10.5.3 Classification Procedure 10.6 Metrics, Experiments and Software 10.6.1 Software. 10.6.2 Experiments. 11 Stochastic Processes 11.1 The Classical Characterization of Stochastic Processes 11.1.1 Basic Definitions 11.1.2 Precise Markov Chains 11.2 Event-driven Random Processes 11.3 Imprecise Markov Chains 11.3.1 From Precise to Imprecise Markov Chains 11.3.2 Imprecise Markov Models under Epistemic Irrelevance. 11.3.3 Imprecise Markov Models Under Strong Independence. 11.3.4 When Does the Interpretation of Independence (not) Matter? 11.4 Limit Behaviour of Imprecise Markov Chains 11.4.1 Metric Properties of Imprecise Probability Models 11.4.2 The Perron-Frobenius Theorem 11.4.3 Invariant Distributions 11.4.4 Coefficients of Ergodicity 11.4.5 Coefficients of Ergodicity for Imprecise Markov Chains. 11.5 Further Reading 12 Financial Risk Measurement 12.1 Introduction 12.2 Imprecise Previsions and Betting 12.3 Imprecise Previsions and Risk Measurement 12.3.1 Risk Measures as Imprecise Previsions 12.3.2 Coherent Risk Measures 12.3.3 Convex Risk Measures (and Previsions) 12.4 Further Reading 13 Engineering 13.1 Introduction 13.2 Probabilistic Dimensioning in a Simple Example 13.3 Random Set Modelling of the Output Variability 13.4 Sensitivity Analysis 13.5 Hybrid Models. 13.6 Reliability Analysis and Decision Making in Engineering 13.7 Further Reading 14 Reliability and Risk 14.1 Introduction 14.2 Stress-strength Reliability 14.3 Statistical Inference in Reliability and Risk 14.4 NPI in Reliablity and Risk 14.5 Discussion and Research Challenges 15 Elicitation 15.1 Methods and Issues 15.2 Evaluating Imprecise Probability Judgements 15.3 Factors Affecting Elicitation 15.4 Further Reading 16 Computation 16.1 Introduction 16.2 Natural Extension 16.2.1 Conditional Lower Previsions with Arbitrary Domains. 16.2.2 The Walley-Pelessoni-Vicig Algorithm 16.2.3 Choquet Integration 16.2.4 Mobius Inverse 16.2.5 Linear-Vacuous Mixture 16.3 Decision Making 16.3.1 Maximin, Maximax, and Hurwicz 16.3.2 Maximality 16.3.3 E-Admissibility 16.3.4 Interval Dominance References Author index Subject index

Introduction to imprecise probabilities

https://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/pdf/gini.pdf

Thomas Augustin

Papers

Conditional variable importance for random forests

Generating survival times to simulate Cox proportional hazards models

Introduction to imprecise probabilities

Unbiased Split Selection for Classification Trees based on the Gini Index

Nonparametric predictive inference and interval probability