Online feedback mechanisms harness the bidirectional communication capabilities of the Internet to engineer large-scale, word-of-mouth networks. Best known so far as a technology for building trust and fostering cooperation in online marketplaces, such as eBay, these mechanisms are poised to have a much wider impact on organizations. Their growing popularity has potentially important implications for a wide range of management activities such as brand building, customer acquisition and retention, product development, and quality assurance. This paper surveys our progress in understanding the new possibilities and challenges that these mechanisms represent. It discusses some important dimensions in which Internet-based feedback mechanisms differ from traditional word-of-mouth networks and surveys the most important issues related to their design, evaluation, and use. It provides an overview of relevant work in game theory and economics on the topic of reputation. It discusses how this body of work is being extended and combined with insights from computer science, management science, sociology, and psychology to take into consideration the special properties of online environments. Finally, it identifies opportunities that this new area presents for operations research/management science (OR/MS) research.

/pdf/the-digitization-of-word-of-mouth-promise-and-challenges-of-1nibi8no0b.pdf

The Digitization of Word of Mouth: Promise and Challenges of Online Feedback Mechanisms

Model-based Bayesian Reinforcement Learning (BRL) provides a principled solution to dealing with the exploration-exploitation trade-off, but such methods typically assume a fully observable environments. The few Bayesian RL methods that are applicable in partially observable domains, such as the Bayes-Adaptive POMDP (BA-POMDP), scale poorly. To address this issue, we introduce the Factored BA-POMDP model (FBA-POMDP), a framework that is able to learn a compact model of the dynamics by exploiting the underlying structure of a POMDP. The FBA-POMDP framework casts the problem as a planning task, for which we adapt the Monte-Carlo Tree Search planning algorithm and develop a belief tracking method to approximate the joint posterior over the state and model variables. Our empirical results show that this method outperforms a number of BRL baselines and is able to learn efficiently when the factorization is known, as well as learn both the factorization and the model parameters simultaneously.

/pdf/bayesian-reinforcement-learning-in-factored-pomdps-xpra8a8w97.pdf

Bayesian Reinforcement Learning in Factored POMDPs

In recent years, there has been an increasing interest in the adoption of emerging sensing technologies for instrumentation within a variety of structural systems. Wireless sensors and sensor networks are emerging as sensing paradigms that the structural engineering field has begun to consider as substitutes for traditional tethered monitoring systems. A benefit of wireless structural monitoring systems is that they are inexpensive to install because extensive wiring is no longer required between sensors and the data acquisition system. Researchers are discovering that wireless sensors are an exciting technology that should not be viewed as simply a substitute for traditional tethered monitoring systems. Rather, wireless sensors can play greater roles in the processing of structural response data; this feature can be utilized to screen data for signs of structural damage. Also, wireless sensors have limitations that require novel system architectures and modes of operation. This paper is intended to serve as a summary review of the collective experience the structural engineering community has gained from the use of wireless sensors and sensor networks for monitoring structural performance and health.

http://www-personal.umich.edu/~jerlynch/papers/SVDReview.pdf

A summary review of wireless sensors and sensor networks for structural health monitoring

Cooperative multi-agent systems (MAS) are ones in which several agents attempt, through their interaction, to jointly solve tasks or to maximize utility. Due to the interactions among the agents, multi-agent problem complexity can rise rapidly with the number of agents or their behavioral sophistication. The challenge this presents to the task of programming solutions to MAS problems has spawned increasing interest in machine learning techniques to automate the search and optimization process. We provide a broad survey of the cooperative multi-agent learning literature. Previous surveys of this area have largely focused on issues common to specific subareas (for example, reinforcement learning, RL or robotics). In this survey we attempt to draw from multi-agent learning work in a spectrum of areas, including RL, evolutionary computation, game theory, complex systems, agent modeling, and robotics. We find that this broad view leads to a division of the work into two categories, each with its own special issues: applying a single learner to discover joint solutions to multi-agent problems (team learning), or using multiple simultaneous learners, often one per agent (concurrent learning). Additionally, we discuss direct and indirect communication in connection with learning, plus open issues in task decomposition, scalability, and adaptive dynamics. We conclude with a presentation of multi-agent learning problem domains, and a list of multi-agent learning resources.

/pdf/cooperative-multi-agent-learning-the-state-of-the-art-4jmhlfb1be.pdf

Cooperative Multi-Agent Learning: The State of the Art

1. Cognitive Architecture 2. The Modular Organization of the Mind 3. Human Associative Memory 4. The Adaptive Control of Thought 5. What Does It Take to Be Human? Lessons From High School Algebra 6. How Can the Human Mind Occur?

How Can the Human Mind Occur in the Physical Universe

The problem of finding and grasping a target object in a cluttered, uncertain environment, target object search, is a common and important problem in robotics. One key challenge is the uncertainty of locating and recognizing each object in a cluttered environment due to noisy perception and occlusions. Furthermore, the uncertainty in localization makes manipulation difficult and uncertain. To cope with these challenges, we formulate the target object search task as a partially observable Markov decision process (POMDP), enabling the robot to reason about perceptual and manipulation uncertainty while searching. To further address the manipulation difficulty, we propose Parameterized Action Partially Observable Monte-Carlo Planning (PA-POMCP), an algorithm that evaluates manipulation actions by taking into account the effect of the robot’s current belief on the success of the action execution. In addition, a novel run-time initial belief generator and a state value estimator are introduced in this paper to facilitate the PA-POMCP algorithm. Our experiments show that our methods solve the target object search task in settings where simpler methods either take more object movements or fail.

Online Planning for Target Object Search in Clutter under Partial Observability

The POMDP is a powerful framework for reasoning under outcome and information uncertainty, but constructing an accurate POMDP model is difficult. Bayes-Adaptive Partially Observable Markov Decision Processes (BA-POMDPs) extend POMDPs to allow the model to be learned during execution. BA-POMDPs are a Bayesian RL approach that, in principle, allows for an optimal trade-off between exploitation and exploration. Unfortunately, BA-POMDPs are currently impractical to solve for any non-trivial domain. In this paper, we extend the Monte-Carlo Tree Search method POMCP to BA-POMDPs and show that the resulting method, which we call BA-POMCP, is able to tackle problems that previous solution methods have been unable to solve. Additionally, we introduce several techniques that exploit the BA-POMDP structure to improve the efficiency of BA-POMCP along with proof of their convergence.

/pdf/learning-in-pomdps-with-monte-carlo-tree-search-4at0v0veft.pdf

Learning in POMDPs with Monte Carlo Tree Search

Many AI systems, such as robots, must plan under time constraints. The most popular search approach in robotics so far is anytime search (Likhachev and Ferguson 2009), in which the algorithm quickly finds a suboptimal plan, and then continues to find better plans as time passes, until eventually converging on an optimal plan. However, the time until the first plan is returned is not controllable, so such methods inherently involve idling the system’s operation before ‘real’ execution can begin. Real-time search methods provide hard real-time bounds on action selection time, yet to our knowledge, they have not yet been demonstrated for robotic systems. In this work, we compare anytime and realtime heuristic search methods in their ability to allow agents to achieve goals quickly. Our results suggest that realtime search is more broadly applicable and often achieves goals faster than anytime search, while anytime search finds shorter plans and does not suffer from dead-ends. Our central performance metric is goal achievement time (GAT), which measures the time starting from when the agent receives a goal specification and begins planning to achieve it and ending when the agent reaches a goal state and is done with the task (Kiesel, Burns, and Ruml 2015). This metric provides a natural way to compare anytime algorithms, which form a complete plan before beginning execution, with on-line algorithms like real-time search. One of the most well-known anytime search algorithms is Anytime Repairing A* (Likhachev, Gordon, and Thrun 2004) which runs a series of weighted A* searches. As soon as the first plan is found, the agent starts moving. However, in order for the previous search effort to remain applicable as the search continues for better plans, the search is actually performed starting at the goal state and searching towards the agent. This means that, even though the agent moves, the g-values (cost to a node from the original goal state) are still useful in the subsequent searches. There are three important limitations of anytime search for concurrent planning and acting. First, searching backward from the goal requires the domain to have a predecessor function, which may be non-trivial. In contrast, real-time search can be used with any domain in which a forward simulator is available. Second, the agent cannot start moving until the first complete plan is formulated. In some systems, a delay in starting execution is merely undesirable, but in others such as fixed-wing aircraft, it is inherently infeasible. Third, because the time it will take for the search to find a new plan is not known in advance, it is not obvious what the goal state for the backwards search should be. In practice, a state along the current plan that is some predefined distance ahead of the agent can be selected, but this is ad hoc. Real-time search algorithms, such as LSS-LRTA* (Koenig and Sun 2009), expand the search space until a hard real-time bound is reached, at which point the agent will commit to one or more steps towards the edge of the expanded search space. The heuristic values of expanded states are then updated by propagating information backward from the lookahead frontier. While the selected actions are being executed, the search computes the next action to take. In this way, the agent iteratively constructs a trajectory. Real-time search suffers from two important limitations. First, it is incomplete in any domain in which there are dead end states from where the goal is unreachable. For example, the agent must be able to look far enough ahead to detect a fatal collision. Second, errors (‘local minima’) in the heuristic function can cause real-time search algorithms to visit the same states many times (‘scrub’), in order to update the heuristic values inside the minimum. If the lookahead is smaller than the size of the minimum, repeated scrubbing can be required (Sturtevant and Bulitko 2014). This behavior appears irrational to an outside observer and is undesirable in many applications, including robotics (Likhachev 2016).

Sammie Katt

Papers

Bayesian Reinforcement Learning in Factored POMDPs

Online Planning for Target Object Search in Clutter under Partial Observability

Learning in POMDPs with Monte Carlo Tree Search

Learning in POMDPs with Monte Carlo Tree Search

Anytime versus Real-Time Heuristic Search for On-Line Planning.