This study integrated multiple nationwide administrative databases and electronic medical records with the four-decade-long Dunedin birth cohort study to test child-to-adult prediction in a different way, using a population-segmentation approach.

Childhood forecasting of a small segment of the population with large economic burden

As we discussed in Part I of this topic, there is a clear desire to model and comprehend human behavior. Given the popular presupposition of human reasoning as the standard for learning and decision-making, there have been significant efforts and a growing trend in research to replicate these innate human abilities in artificial systems. In Part I, we discussed learning methods which generate a model of behavior from exploration of the system and feedback based on the exhibited behavior as well as topics relating to the use of or accounting for beliefs with respect to applicable skills or mental states of others. In this work, we will continue the discussion from the perspective of methods which focus on the assumed cognitive abilities, limitations, and biases demonstrated in human reasoning. We will arrange these topics as follows (i) methods such as cognitive architectures, cognitive heuristics, and related which demonstrate assumptions of limitations on cognitive resources and how that impacts decisions and (ii) methods which generate and utilize representations of bias or uncertainty to model human decision-making or the future outcomes of decisions.

Modeling Human Behavior Part II - Cognitive approaches and Uncertainty

Healthy adults flexibly adapt their learning strategies to ongoing changes in uncertainty, a key feature of adaptive behaviour. However, the developmental trajectory of this ability is yet unknown, as developmental studies have not incorporated trial-to-trial variation in uncertainty in their analyses or models. To address this issue, we compared adolescents’ and adults’ trial-to-trial dynamics of uncertainty, learning rate, and exploration in two tasks that assess learning in noisy but otherwise stable environments. In an estimation task—which provides direct indices of trial-specific learning rate—both age groups reduced their learning rate over time, as self-reported uncertainty decreased. Accordingly, the estimation data in both groups was better explained by a Bayesian model with dynamic learning rate (Kalman filter) than by conventional reinforcement-learning models. Furthermore, adolescents’ learning rates asymptoted at a higher level, reflecting an over-weighting of the most recent outcome, and the estimated Kalman-filter parameters suggested that this was due to an overestimation of environmental volatility. In a choice task, both age groups became more likely to choose the higher-valued option over time, but this increase in choice accuracy was smaller in the adolescents. In contrast to the estimation task, we found no evidence for a Bayesian expectation-updating process in the choice task, suggesting that estimation and choice tasks engage different learning processes. However, our modeling results of the choice task suggested that both age groups reduced their degree of exploration over time, and that the adolescents explored overall more than the adults. Finally, age-related differences in exploration parameters from fits to the choice data were mediated by participants’ volatility parameter from fits to the estimation data. Together, these results suggest that adolescents overestimate the rate of environmental change, resulting in elevated learning rates and increased exploration, which may help understand developmental changes in learning and decision-making.

Uncertainty-driven regulation of learning and exploration in adolescents

Many psychiatric disorders are marked by impaired decision-making during an approach-avoidance conflict. Current experiments elicit approachavoidance conflicts in bandit tasks by pairing an individual’s actions with consequences that are simultaneously desirable (reward) and undesirable (harm). We frame approach-avoidance conflict tasks as a multi-objective multi-armed bandit. By defining a general decision-maker as a limiting sequence of actions, we disentangle the decision process from learning. Each decision maker can then be identified as a multi-dimensional point representing its long-term average expected outcomes, while different decision making models can be associated by the geometry of their ‘feasible region’, the set of all possible long term performances on a fixed task. We introduce three example decision-makers based on popular reinforcement learning models and characterize their feasible regions, including whether they can be Pareto optimal. From this perspective, we find that existing tasks are unable to distinguish between the three examples of decision-makers. We show how to design new tasks whose geometric structure can be used to better distinguish between decision-makers. These findings are expected to guide the design of approach-avoidance conflict tasks and the modeling of resulting decision-making behavior.

Placing approach-avoidance conflict within the framework of multi-objective reinforcement learning

Satisfying a variety of conflicting needs in a changing environment is a fundamental challenge for any adaptive agent. Here, we show that designing an agent in a modular fashion as a collection of subagents, each dedicated to a separate need, powerfully enhanced the agent’s capacity to satisfy its overall needs. We used the formalism of deep reinforcement learning to investigate a biologically relevant multi-objective task: continually maintaining homeostasis of a set of physiologic variables. We then conducted simulations in a variety of environments and compared how modular agents performed relative to standard monolithic agents (i.e., agents that aimed to satisfy all needs in an integrated manner using a single aggregate measure of success). Simulations revealed that modular agents: a) exhibited a form of exploration that was intrinsic and emergent rather than extrinsically imposed; b) were robust to changes in non-stationary environments, and c) scaled gracefully in their ability to maintain home-ostasis as the number of conflicting objectives increased. Supporting analysis suggested that the robustness to changing environments and increasing numbers of needs were due to intrinsic exploration and efficiency of representation afforded by the modular architecture. These results suggest that the normative principles by which agents have adapted to complex changing environments may also explain why humans have long been described as consisting of ‘multiple selves’. Significance Statement Adaptive agents must continually satisfy a range of distinct and possibly conflicting needs. In most models of learning, a monolithic agent tries to maximize one value that measures how well it balances its needs. However, this task is difficult when the world is changing and needs are many. Here, we considered an agent as a collection of modules each dedicated to a particular need and competing for control of action. Compared to the standard monolithic approach, modular agents were much better at maintaining homeostasis of a set of internal variables in simulated environments, both static and changing. These results suggest that having ‘multiple selves’ may represent an evolved solution to the universal problem of balancing multiple needs in changing environments.

https://www.biorxiv.org/content/biorxiv/early/2022/12/17/2022.12.16.520795.full.pdf

Having “multiple selves” helps learning agents explore and adapt in complex changing worlds

Recent experiments and theories of human decision-making suggest positive and negative errors are processed and encoded differently by serotonin and dopamine, with serotonin possibly serving to oppose dopamine and protect against risky decisions. We introduce a temporal difference (TD) model of human decision-making to account for these features. Our model involves two critics, an optimistic learning system and a pessimistic learning system, whose predictions are integrated in time to control how potential decisions compete to be selected. Our model predicts that human decision-making can be decomposed along two dimensions: the degree to which the individual is sensitive to (1) risk and (2) uncertainty. In addition, we demonstrate that the model can learn about the mean and standard deviation of rewards, and provide information about reaction time despite not modeling these variables directly. Lastly, we simulate a recent experiment to show how updates of the two learning systems could relate to dopamine and serotonin transients, thereby providing a mathematical formalism to serotonin’s hypothesized role as an opponent to dopamine. This new model should be useful for future experiments on human decision-making.

/pdf/a-competition-of-critics-in-human-decision-making-35liqlcast.pdf

A Competition of Critics in Human Decision-Making

Recent experiments and theories of human decision-making suggest positive and negative errors are processed and encoded differently by serotonin and dopamine, with serotonin possibly serving to oppose dopamine and protect against risky decisions. We introduce a temporal difference (TD) model of human decision-making to account for these features. Our model involves two opposing counsels, an optimistic learning system and a pessimistic learning system, whose predictions are integrated in time to control how potential decisions compete to be selected. Our model predicts that human decision-making can be decomposed along two dimensions: the degree to which the individual is sensitive to (1) risk and (2) uncertainty. In addition, we demonstrate that the model can learn about reward expectations and uncertainty, and provide information about reaction time despite not modeling these variables directly. Lastly, we simulate a recent experiment to show how updates of the two learning systems could relate to dopamine and serotonin transients, thereby providing a mathematical formalism to serotonin9s hypothesized role as an opponent to dopamine. This new model should be useful for future experiments on human decision-making.

/pdf/a-competition-of-critics-in-human-decision-making-3fixkdfu2v.pdf

A competition of critics in human decision-making

/pdf/a-model-of-opposing-counsels-in-human-decision-making-2n818viyy6.pdf

A model of opposing counsels in human decision-making

At the onset of sensory stimulation, the variability and co-variability of spiking activity is widely reported to decrease, especially in cortex. Considering the potential benefits of such decreased variability for coding, it has been suggested that this could be a general principle governing all sensory systems. Here we show that this is not so. We recorded neurons in olfactory bulb (OB) of anesthetized rats and found increased variability and co-variability of spiking at the onset of olfactory stimulation. Using models and analysis, we predicted that these increases arise due to network interactions within OB, without increasing variability of input from the nose. We tested and confirmed this prediction using optogenetic stimulation of OB in awake animals. Our results establish increases in spiking variability at stimulus onset as a viable alternative coding strategy to the more commonly observed decreases in variability in many cortical systems.

https://www.biorxiv.org/content/biorxiv/early/2021/03/10/2021.03.05.434131.full.pdf

Odor-evoked Increases in Spiking Variability in the Olfactory Bulb

The majority of olfaction studies focus on orthonasal stimulation where odors enter via the front nasal cavity, while retronasal olfaction, where odors enter the rear of the nasal cavity during feeding, is understudied. The processing of retronasal odors via coordinated spiking of neurons in the olfactory bulb (OB) is largely unknown. To this end, we use multi-electrode array in vivo recordings of rat OB mitral cells (MC) in response to a food odor with both modes of stimulation, and find significant differences in evoked firing rates and spike count covariances (i.e., noise correlations). To better understand these differences, we develop a single-compartment biophysical OB model that is able to reproduce key properties of important OB cell types. Prior experiments in olfactory receptor neurons (ORN) showed retro stimulation yields slower and spatially smaller ORN inputs than with ortho, yet whether this is consequential for OB activity remains unknown. Indeed with these specifications for ORN inputs, our OB model captures the trends in our OB data. We also analyze how first and second order ORN input statistics dynamically transfer to MC spiking statistics with a phenomenological linear-nonlinear filter model, and find that retro inputs result in larger temporal filters than ortho inputs. Finally, our models show that the temporal profile of ORN is crucial for capturing our data and is thus a distinguishing feature between ortho and retro stimulation, even at the OB. Using data-driven modeling, we detail how ORN inputs result in differences in OB dynamics and MC spiking statistics. These differences may ultimately shape how ortho and retro odors are coded. Author summary Olfaction is a key sense for many cognitive and behavioral tasks, and is particularly unique because odors can naturally enter the nasal cavity from the front or rear, i.e., ortho- and retro-nasal, respectively. Yet little is known about the differences in coordinated spiking in the olfactory bulb with ortho versus retro stimulation, let alone how these different modes of olfaction may alter coding of odors. We simultaneously record many cells in rat olfactory bulb to assess the differences in spiking statistics, and develop a biophysical olfactory bulb network model to study the reasons for these differences. Using theoretical and computational methods, we find that the olfactory bulb transfers input statistics differently for retro stimulation relative to ortho stimulation. Furthermore, our models show that the temporal profile of inputs is crucial for capturing our data and is thus a distinguishing feature between ortho and retro stimulation, even at the olfactory bulb. Understanding the spiking dynamics of the olfactory bulb with both ortho and retro stimulation is a key step for ultimately understanding how the brain codes odors with different modes of olfaction.

/pdf/analyzing-the-differences-in-olfactory-bulb-mitral-cell-53lvgr4lyx.pdf

Cheng Ly

Papers

A Competition of Critics in Human Decision-Making

A competition of critics in human decision-making

A model of opposing counsels in human decision-making

Odor-evoked Increases in Spiking Variability in the Olfactory Bulb

Analyzing the differences in olfactory bulb mitral cell spiking with ortho- and retronasal stimulation