Modelling for Prediction vs. Modelling for Understanding: Commentary on Musso et al. (2013)
19 Dec 2013-Vol. 1, Iss: 2, pp 99-101
TL;DR: It is concluded that ANNs have high potential for theoretical and practical improvements in learning sciences and researchers in the learning sciences should prefer more theory-driven and parsimonious modelling techniques over ANNs whenever possible.
Abstract: Musso et al. (2013) predict students’ academic achievement with high accuracy one year in advance from cognitive and demographic variables, using artificial neural networks (ANNs). They conclude that ANNs have high potential for theoretical and practical improvements in learning sciences. ANNs are powerful statistical modelling tools but they can mainly be used for exploratory modelling. Moreover, the output generated from ANNs cannot be fully translated into a meaningful set of rules because they store information about input-output relations in a complex, distributed, and implicit way. These problems hamper systematic theory-building as well as communication and justification of model predictions in practical contexts. Modern-day regression techniques, including (Bayesian) structural equation models, have advantages similar to those of ANNs but without the drawbacks. They are able to handle numerous variables, non-linear effects, multi-way interactions, and incomplete data. Thus, researchers in the learning sciences should prefer more theory-driven and parsimonious modelling techniques over ANNs whenever possible.
Citations
More filters
08 Jul 2018
TL;DR: This paper investigates the use of a hybrid model comprising multiple artificial neural networks with a final C4.5 decision tree classifier to investigate the potential of explaining the classification decision through production rules and the significant tree size questions the rule transparency to a human.
Abstract: The Artificial Neural Network is generally considered to be an effective classifier, but also a “Black Box” component whose internal behavior cannot be understood by human users. This lack of transparency forms a barrier to acceptance in high-stakes applications by the general public. This paper investigates the use of a hybrid model comprising multiple artificial neural networks with a final C4.5 decision tree classifier to investigate the potential of explaining the classification decision through production rules. Two large datasets collected from comprehension studies are used to investigate the value of the C4.5 decision tree as the overall comprehension classifier in terms of accuracy and decision transparency. Empirical trials show that higher accuracies are achieved through using a decision tree classifier, but the significant tree size questions the rule transparency to a human.
7 citations
Cites background from "Modelling for Prediction vs. Modell..."
...[11] observed that the basic problems of communicating how they reach their conclusions in meaningful terms has yet to be solved....
[...]
30 Jan 2015
TL;DR: Two articles, Edelsbrunner and, Schneider (2013), and Nokelainen and Silander (2014) comment on Musso, Kyndt, Cascallar, and Dochy, with a perspective on its place among other predictive approaches.
Abstract: Two articles, Edelsbrunner and, Schneider (2013), and Nokelainen and Silander (2014) comment on Musso, Kyndt, Cascallar, and Dochy (2013). Several relevant issues are raised and some important clarifications are made in response to both commentaries. Predictive systems based on artificial neural networks continue to be the focus of current research and several advances have improved the model building and the interpretation of the resulting neural network models. What is needed is the courage and open-mindedness to actually explore new paths and rigorously apply new methodologies which can perhaps, sometimes unexpectedly, provide new conceptualisations and tools for theoretical advancement and practical applied research. This is particularly true in the fields of educational science and social sciences, where the complexity of the problems to be solved requires the exploration of proven methods and new methods, the latter usually not among the common arsenal of tools of neither practitioners nor researchers in these fields. This response will enrich the understanding of the predictive systems methodology proposed by the authors and clarify the application of the procedure, as well as give a perspective on its place among other predictive approaches.
7 citations
Cites background or methods or result from "Modelling for Prediction vs. Modell..."
...Edelsbrunner and Schneider (2013) in their commentary on Musso, Kyndt, Cascallar and Dochy (2013) argue that artificial neural networks (ANNs) should only be used as exploratory modelling techniques, in spite of being powerful statistical modelling tools with demonstrated ability to improve…...
[...]
...The reasons Edelsbrunner and Schneider (2013) argue for their rather strong position are centred on two main arguments: (a) that the output from ANNs cannot be fully translated into a meaningful set of rules because of a lack of accessibility to the input-output relationships, and (b) that there is…...
[...]
...Therefore, contrary to what has been pointed out by Edelsbrunner and Schneider (2013) and quoted by Golino and Gomes (2014), the ANN approach offers the potential to examine the complex relationships amongst its components....
[...]
...Now it 69 | F L R expressed in Edelsbrunner & Schneider (2013)....
[...]
...The second main argument regarding problems associated with the ANN methodology, as claimed by Edelsbrunner and Schneider (2013), has to do with the lack of some statistical parameters in ANNs....
[...]
TL;DR: It is suggested that machine learning remains one of the promising forecasting technologies with the power to enhance effective academic forecasting that would assist the education industry in planning and making better decisions to enrich the quality of education.
Abstract: The study examines the prospects and challenges of machine learning (ML) applications in academic forecasting. Predicting academic activities through machine learning algorithms presents an enhanced means to accurately forecast academic events, including the academic performances and the learning style of students. The use of machine learning algorithms such as K-nearest neighbor (KNN), random forest, bagging, artificial neural network (ANN), and Bayesian neural network (BNN) has potentials that are currently being applied in the education sector to predict future events. Many gaps in the traditional forecasting techniques have greatly been bridged by the use of artificial intelligence-based machine learning algorithms thereby aiding timely decision-making by education stakeholders. ML algorithms are deployed by educational institutions to predict students' learning behaviours and academic achievements, thereby giving them the opportunity to detect at-risk students early and then develop strategies to help them overcome their weaknesses. However, despite the benefits associated with the ML approach, there exist some limitations that could affect its correctness or deployment in forecasting academic events, e.g., proneness to errors, data acquisition, and time-consuming issues. Nonetheless, we suggest that machine learning remains one of the promising forecasting technologies with the power to enhance effective academic forecasting that would assist the education industry in planning and making better decisions to enrich the quality of education.
3 citations
01 Jan 2017
TL;DR: Zusammenfassung et al. as mentioned in this paper discussed psychometric issues in Research on Scientific Reasoning and proposed a methodological approach to find the root cause of such issues.
Abstract: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I Zusammenfassung . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III 1 General Introduction 1 1.1 History of Research on Scientific Thinking . . . . . . . . . . . . . . . . . . . . . . 4 1.2 The Present Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3 Methodological Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2 Psychometric Issues in Research on Scientific Reasoning 35 2.
3 citations
28 Apr 2014
TL;DR: In this paper, the authors discuss issues related to model fitting, comparison of classification accuracy of generative and discriminative models, and two (or more) cultures of data modeling.
Abstract: This commentary to the recent article by Musso et al. (2013) discusses issues related to model fitting, comparison of classification accuracy of generative and discriminative models, and two (or more) cultures of data modeling. We start by questioning the extremely high classification accuracy with an empirical data from a complex domain. There is a risk that we model perfect nonsense perfectly. Our second concern is related to the relevance of comparing multilayer perceptron neural networks and linear discriminant analysis classification accuracy indices. We find this problematic, as it is like comparing apples and oranges. It would have been easier to interpret the model and the variable (group) importance’s if the authors would have compared MLP to some discriminative classifier, such as group lasso logistic regression. Finally, we conclude our commentary with a discussion about the predictive properties of the adopted data modeling approach.
1 citations
References
More filters
23 Aug 2013
TL;DR: In this article, the authors used cognitive and non-cognitive measures of students, together with background information, in order to design predictive models of student performance using artificial neural networks (ANN).
Abstract: Many studies have explored the contribution of different factors from diverse theoretical perspectives to the explanation of academic performance. These factors have been identified as having important implications not only for the study of learning processes, but also as tools for improving curriculum designs, tutorial systems, and students’ outcomes. Some authors have suggested that traditional statistical methods do not always yield accurate predictions and/or classifications (Everson, 1995; Garson, 1998). This paper explores a relatively new methodological approach for the field of learning and education, but which is widely used in other areas, such as computational sciences, engineering and economics. This study uses cognitive and non-cognitive measures of students, together with background information, in order to design predictive models of student performance using artificial neural networks (ANN). These predictions of performance constitute a true predictive classification of academic performance over time, a year in advance of the actual observed measure of academic performance. A total sample of 864 university students of both genders, ages ranging between 18 and 25 was used. Three neural network models were developed. Two of the models (identifying the top 33% and the lowest 33% groups, respectively) were able to reach 100% correct identification of all students in each of the two groups. The third model (identifying low, mid and high performance levels) reached precisions from 87% to 100% for the three groups. Analyses also explored the predicted outcomes at an individual level, and their correlations with the observed results, as a continuous variable for the whole group of students. Results demonstrate the greater accuracy of the ANN compared to traditional methods such as discriminant analyses. In addition, the ANN provided information on those predictors that best explained the different levels of expected performance. Thus, results have allowed the identification of the specific influence of each pattern of variables on different levels of academic performance, providing a better understanding of the variables with the greatest impact on individual learning processes, and of those factors that best explain these processes for different academic levels.
56 citations
01 May 2006
TL;DR: The book traces the development of this methodology and demonstrates how it opens up new ways of thinking about traditional problems, and academic researchers will gain a design template for studying both the linear and non-linear elements of a given problem, and thus enhance their own research.
Abstract: While the term neural networks may be unfamiliar to many organizational psychologists, exciting new applications of artificial intelligence are attracting notice among organizational behavior researchers. In "Neural Networks in Organizational Research: Applying Pattern Recognition to the Analysis of Organizational Behavior", authors David Scarborough and Mark Somers bring researchers, academics, and practitioners up to speed on this emerging field, in which powerful computing capabilities offer new insights into longstanding, complex I/O questions such as employee selection and behavioral prediction. Neural networks mimic the way the human brain works, using interconnected nodes and feedback loops to "learn" to recognize even subtle patterns in vast amounts of data. They can process data far more quickly and efficiently than conventional techniques can, and produce better empirical results. They are especially useful for modeling nonlinear processes. The book traces the development of this methodology and demonstrates how it opens up new ways of thinking about traditional problems. Academic researchers will gain a design template for studying both the linear and non-linear elements of a given problem, and thus enhance their own research.
47 citations
"Modelling for Prediction vs. Modell..." refers background or methods in this paper
...While one can assess how well an ANN works, it is difficult to comprehensively explain why it performs well or not (Scarborough & Somers, 2006)....
[...]
...First, the construction of ANN models such as those used by Musso et al. is highly explorative apart from choosing relevant input and output variables (Günther, Pigeot, &Bammann, 2012; Scarborough & Somers, 2006)....
[...]
TL;DR: This study shows that neural networks are a promising approach for analyzing gene-environment interactions, if no prior knowledge of the correct nature of the relationship between co-variables and response variable is present, and provide a valuable alternative to regression methods that are limited to the analysis of linearly separable data.
Abstract: Gene-environment interactions play an important role in the etiological pathway of complex diseases. An appropriate statistical method for handling a wide variety of complex situations involving interactions between variables is still lacking, especially when continuous variables are involved. The aim of this paper is to explore the ability of neural networks to model different structures of gene-environment interactions. A simulation study is set up to compare neural networks with standard logistic regression models. Eight different structures of gene-environment interactions are investigated. These structures are characterized by penetrance functions that are based on sigmoid functions or on combinations of linear and non-linear effects of a continuous environmental factor and a genetic factor with main effect or with a masking effect only. In our simulation study, neural networks are more successful in modeling gene-environment interactions than logistic regression models. This outperfomance is especially pronounced when modeling sigmoid penetrance functions, when distinguishing between linear and nonlinear components, and when modeling masking effects of the genetic factor. Our study shows that neural networks are a promising approach for analyzing gene-environment interactions. Especially, if no prior knowledge of the correct nature of the relationship between co-variables and response variable is present, neural networks provide a valuable alternative to regression methods that are limited to the analysis of linearly separable data.
4 citations