The Importance of Falsification in Computational Cognitive Modeling.

TLDR: It is argued here that the simulation of candidate models is necessary to falsify models and therefore support the specific claims about cognitive function made by the vast majority of model-based studies.

About: This article is published in Trends in Cognitive Sciences.The article was published on 2017-06-01 and is currently open access. It has received 293 citations till now. The article focuses on the topics: Cognitive model.

Figures

Figure 2: possible relationships between relative model comparison and model simulation comparison

Figure 1: the exponential increase of computational model-based cognitive neuroscience. The curves on the left show the relative frequency of PubMed entries for “cognitive” (in red) and “cognitive and computational” (in blue) as a function of the year (from the 1970s to the 2014). Their frequencies are calculated relative to the number of entries of 2014, which are therefore defined at 1 for both curves. The bars on the left represent the estimated annual growth of the best fitting exponential curve.

Table 1: percentage of studies as a function of their model-based analytical procedure. “Computational Model” = studies report a computational model-based analysis. “Relative comparison” = studies report a model selection step implicating relative model comparison criteria, such as AIC, BIC or similar. “Best simulation” = studies report the model simulation of the best model, according to relative model selection. “Rival simulation” = studies report the simulation of the best and the rival(s) models (note that the presence of rival model simulation represents no guarantee that any statistical analysis is then performed to quantitatively assess the “similarity” of the model simulations to the actual data). The complete list of the studies included in the meta-analysis is available online).

Frequently asked questions

Q1. What are the contributions mentioned in the paper "Computational cognitive neuroscience: model fitting should not replace model simulation" ?

Here the authors argue that the analysis of model simulations is often necessary to support the specific claims about behavioral function that most of model-based studies make. The authors defend this argument both informally by providing a large-scale ( N > 300 ) review of recent studies, and formally by showing how model simulations are necessary to interpret model comparison results. Finally, the authors propose guidelines for future work, which combine model comparison and simulation. 

Q2. What is the way to determine the quality of fit of the models?

4. Fit the competing computational models to the data, in order to obtain, for each model an estimation of the best fitting model parameters and an approximation of the model evidence, that trades-off the quality of fit and model complexity. 

Q3. What are the two main reasons why models are not appropriate to falsify?

Relative model comparison criteria (i.e. various approximations of the model evidence, such as BIC, AIC,) are not appropriate to falsify models because they do not capture certain features of the fitted data: 1) they focus on the evidence in favor of the best, instead of evidence against the rival model, and 2) they are blind to the capacity of tested models to reproduce (or not) any particular phenomenon of interest. 

Q4. What is the learning curve for the task?

The learning curves represent data simulated from this task with a standard RL algorithm (in grey; “No modulation” case) and a model that uses a higher learning rate in the volatile compared to the stable phase (in blue; the “Modulation” case). 

Q5. What is the epistemological specificity of a computational modeling approach?

In natural sciences, it has been proposed that the epistemological specificity of a computational modeling approach, compared to a model-free one, is that, the latter investigates directly the natural phenomenon of interest, whereas the former builds an artificial representation of the natural system (model) and study its behavior22. 

Q6. What is the way to ensure that the two models are not competing?

Simulate ex ante the two (or more) competing computational theories across a large range of parameters (sometimes called a ‘parameter recovery’ procedure) in order to ensure that the task allows the discrimination of the two models (i.e. their model predictions diverge in front of a key experimental manipulation). 

Q7. What is the precept of Occam’s lex parsimoniaea?

In short, this precept dictates that amongst “equally good” explanation of data, the less complex should be held as more likely to be true. 

Q8. What is the definition of computational modeling?

In cognitive neuroscience computational models can also be simply used as tools to quantify different features of the behavioral or neural activity. 

Q9. What is the definition of a model recovery procedure?

Such procedure (that can be defined “model recovery”) would consist in simulating two datasets with two different models and verify (for a given set of models and task specification) which relative model comparison criterion avoid both over- and under-fitting17.