Modelling for Prediction vs. Modelling for Understanding: Commentary on Musso et al. (2013)
19 Dec 2013-Vol. 1, Iss: 2, pp 99-101
TL;DR: It is concluded that ANNs have high potential for theoretical and practical improvements in learning sciences and researchers in the learning sciences should prefer more theory-driven and parsimonious modelling techniques over ANNs whenever possible.
Abstract: Musso et al. (2013) predict students’ academic achievement with high accuracy one year in advance from cognitive and demographic variables, using artificial neural networks (ANNs). They conclude that ANNs have high potential for theoretical and practical improvements in learning sciences. ANNs are powerful statistical modelling tools but they can mainly be used for exploratory modelling. Moreover, the output generated from ANNs cannot be fully translated into a meaningful set of rules because they store information about input-output relations in a complex, distributed, and implicit way. These problems hamper systematic theory-building as well as communication and justification of model predictions in practical contexts. Modern-day regression techniques, including (Bayesian) structural equation models, have advantages similar to those of ANNs but without the drawbacks. They are able to handle numerous variables, non-linear effects, multi-way interactions, and incomplete data. Thus, researchers in the learning sciences should prefer more theory-driven and parsimonious modelling techniques over ANNs whenever possible.
Citations
More filters
08 Jul 2018
TL;DR: This paper investigates the use of a hybrid model comprising multiple artificial neural networks with a final C4.5 decision tree classifier to investigate the potential of explaining the classification decision through production rules and the significant tree size questions the rule transparency to a human.
Abstract: The Artificial Neural Network is generally considered to be an effective classifier, but also a “Black Box” component whose internal behavior cannot be understood by human users. This lack of transparency forms a barrier to acceptance in high-stakes applications by the general public. This paper investigates the use of a hybrid model comprising multiple artificial neural networks with a final C4.5 decision tree classifier to investigate the potential of explaining the classification decision through production rules. Two large datasets collected from comprehension studies are used to investigate the value of the C4.5 decision tree as the overall comprehension classifier in terms of accuracy and decision transparency. Empirical trials show that higher accuracies are achieved through using a decision tree classifier, but the significant tree size questions the rule transparency to a human.
7 citations
Cites background from "Modelling for Prediction vs. Modell..."
...[11] observed that the basic problems of communicating how they reach their conclusions in meaningful terms has yet to be solved....
[...]
30 Jan 2015
TL;DR: Two articles, Edelsbrunner and, Schneider (2013), and Nokelainen and Silander (2014) comment on Musso, Kyndt, Cascallar, and Dochy, with a perspective on its place among other predictive approaches.
Abstract: Two articles, Edelsbrunner and, Schneider (2013), and Nokelainen and Silander (2014) comment on Musso, Kyndt, Cascallar, and Dochy (2013). Several relevant issues are raised and some important clarifications are made in response to both commentaries. Predictive systems based on artificial neural networks continue to be the focus of current research and several advances have improved the model building and the interpretation of the resulting neural network models. What is needed is the courage and open-mindedness to actually explore new paths and rigorously apply new methodologies which can perhaps, sometimes unexpectedly, provide new conceptualisations and tools for theoretical advancement and practical applied research. This is particularly true in the fields of educational science and social sciences, where the complexity of the problems to be solved requires the exploration of proven methods and new methods, the latter usually not among the common arsenal of tools of neither practitioners nor researchers in these fields. This response will enrich the understanding of the predictive systems methodology proposed by the authors and clarify the application of the procedure, as well as give a perspective on its place among other predictive approaches.
7 citations
Cites background or methods or result from "Modelling for Prediction vs. Modell..."
...Edelsbrunner and Schneider (2013) in their commentary on Musso, Kyndt, Cascallar and Dochy (2013) argue that artificial neural networks (ANNs) should only be used as exploratory modelling techniques, in spite of being powerful statistical modelling tools with demonstrated ability to improve…...
[...]
...The reasons Edelsbrunner and Schneider (2013) argue for their rather strong position are centred on two main arguments: (a) that the output from ANNs cannot be fully translated into a meaningful set of rules because of a lack of accessibility to the input-output relationships, and (b) that there is…...
[...]
...Therefore, contrary to what has been pointed out by Edelsbrunner and Schneider (2013) and quoted by Golino and Gomes (2014), the ANN approach offers the potential to examine the complex relationships amongst its components....
[...]
...Now it 69 | F L R expressed in Edelsbrunner & Schneider (2013)....
[...]
...The second main argument regarding problems associated with the ANN methodology, as claimed by Edelsbrunner and Schneider (2013), has to do with the lack of some statistical parameters in ANNs....
[...]
TL;DR: It is suggested that machine learning remains one of the promising forecasting technologies with the power to enhance effective academic forecasting that would assist the education industry in planning and making better decisions to enrich the quality of education.
Abstract: The study examines the prospects and challenges of machine learning (ML) applications in academic forecasting. Predicting academic activities through machine learning algorithms presents an enhanced means to accurately forecast academic events, including the academic performances and the learning style of students. The use of machine learning algorithms such as K-nearest neighbor (KNN), random forest, bagging, artificial neural network (ANN), and Bayesian neural network (BNN) has potentials that are currently being applied in the education sector to predict future events. Many gaps in the traditional forecasting techniques have greatly been bridged by the use of artificial intelligence-based machine learning algorithms thereby aiding timely decision-making by education stakeholders. ML algorithms are deployed by educational institutions to predict students' learning behaviours and academic achievements, thereby giving them the opportunity to detect at-risk students early and then develop strategies to help them overcome their weaknesses. However, despite the benefits associated with the ML approach, there exist some limitations that could affect its correctness or deployment in forecasting academic events, e.g., proneness to errors, data acquisition, and time-consuming issues. Nonetheless, we suggest that machine learning remains one of the promising forecasting technologies with the power to enhance effective academic forecasting that would assist the education industry in planning and making better decisions to enrich the quality of education.
3 citations
01 Jan 2017
TL;DR: Zusammenfassung et al. as mentioned in this paper discussed psychometric issues in Research on Scientific Reasoning and proposed a methodological approach to find the root cause of such issues.
Abstract: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I Zusammenfassung . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III 1 General Introduction 1 1.1 History of Research on Scientific Thinking . . . . . . . . . . . . . . . . . . . . . . 4 1.2 The Present Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3 Methodological Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2 Psychometric Issues in Research on Scientific Reasoning 35 2.
3 citations
28 Apr 2014
TL;DR: In this paper, the authors discuss issues related to model fitting, comparison of classification accuracy of generative and discriminative models, and two (or more) cultures of data modeling.
Abstract: This commentary to the recent article by Musso et al. (2013) discusses issues related to model fitting, comparison of classification accuracy of generative and discriminative models, and two (or more) cultures of data modeling. We start by questioning the extremely high classification accuracy with an empirical data from a complex domain. There is a risk that we model perfect nonsense perfectly. Our second concern is related to the relevance of comparing multilayer perceptron neural networks and linear discriminant analysis classification accuracy indices. We find this problematic, as it is like comparing apples and oranges. It would have been easier to interpret the model and the variable (group) importance’s if the authors would have compared MLP to some discriminative classifier, such as group lasso logistic regression. Finally, we conclude our commentary with a discussion about the predictive properties of the adopted data modeling approach.
1 citations
References
More filters
TL;DR: This interpretation of neural networks is built with fuzzy rules using a new fuzzy logic operator which is defined after introducing the concept of f-duality and offers an automated knowledge acquisition procedure.
Abstract: Artificial neural networks are efficient computing models which have shown their strengths in solving hard problems in artificial intelligence. They have also been shown to be universal approximators. Notwithstanding, one of the major criticisms is their being black boxes, since no satisfactory explanation of their behavior has been offered. In this paper, we provide such an interpretation of neural networks so that they will no longer be seen as black boxes. This is stated after establishing the equality between a certain class of neural nets and fuzzy rule-based systems. This interpretation is built with fuzzy rules using a new fuzzy logic operator which is defined after introducing the concept of f-duality. In addition, this interpretation offers an automated knowledge acquisition procedure.
488 citations
TL;DR: A new visualization approach based on a Sensitivity Analysis (SA) to extract human understandable knowledge from supervised learning black box data mining models, such as Neural Networks, Support Vector Machines and ensembles, including Random Forests.
347 citations
"Modelling for Prediction vs. Modell..." refers methods in this paper
...Recent research has attempted to increase the interpretability of ANNs, for example with the help of visualizations for complex interactions (e.g., Cortez & Embrechts, 2013; Intrator & Intrator, 2001)....
[...]
TL;DR: The approach advocated in this article allows one to determine the extent of sample size sensitivity and the effects of specification error by relying on existing statistical theory underlying covariance structure models.
Abstract: The purpose of this article is to present a strategy for the evaluation and modification of covariance structure models. The approach makes use of recent developments in estimation under non-standard conditions and unified asymptotic theory related to hypothesis testing. Factors affecting the evaluation and modification of these models are reviewed in terms of nonnormality, missing data, specification error, and sensitivity to large sample size. Alternative model evaluation and specification error search strategies are also reviewed. The approach to covariance structure modeling advocated in this article utilizes the LISREL modification index for assessing statistical power, and the expected parameter change statistic for guiding specification error searches. It is argued that the common approach of utilizing alternative fit indices does not allow the investigator to rule out plausible explanations for model misfit. The approach advocated in this article allows one to determine the extent of sample size sensitivity and the effects of specification error by relying on existing statistical theory underlying covariance structure models.
255 citations
"Modelling for Prediction vs. Modell..." refers background in this paper
...Researchers can compare competing theories and advance assumptions that are not in line with the empirical data by fitting a series of statistical models that differ in theoretically relevant aspects (Kaplan, 1990)....
[...]
Book•
17 Sep 2012
TL;DR: Basic and Advanced Bayesian Structural Equation Modeling introduces basic and advanced SEMs for analyzing various kinds of complex data, such as ordered and unordered categorical data, multilevel data, mixture data, longitudinal data, highly non-normal data, as well as some of their combinations.
Abstract: Basic and Advanced Bayesian Structural Equation Modeling introduces basic and advanced SEMs for analyzing various kinds of complex data, such as ordered and unordered categorical data, multilevel data, mixture data, longitudinal data, highly non-normal data, as well as some of their combinations. In addition, Bayesian semiparametric SEMs to capture the true distribution of explanatory latent variables are introduced, whilst SEM with a nonparametric structural equation to assess unspecified functional relationships among latent variables are also explored.
168 citations
"Modelling for Prediction vs. Modell..." refers background in this paper
...Especially Bayesian structural equation models represent a strong advancement in modelling non-linear relations, assessing unspecified relations and handling highly non-normal and hierarchical data (Song & Lee, 2012)....
[...]
TL;DR: In this article, the authors introduce a method that demonstrates the effects of inputs on output of ANNs by using novel robustification techniques, and conclude that neural networks estimated with sufficient regularization can be reliably interpreted using the method presented in this paper.
83 citations
"Modelling for Prediction vs. Modell..." refers methods in this paper
...Recent research has attempted to increase the interpretability of ANNs, for example with the help of visualizations for complex interactions (e.g., Cortez & Embrechts, 2013; Intrator & Intrator, 2001)....
[...]