This paper presents the application of a continuous space Monte Carlo tree search (MCTS) technique to cognitive tracking radar. Previous cognitive tracking radar implemen-tations have used approximate dynamic programming (ADP) techniques like receding horizon (RH) and policy rollout to intel-ligently select waveforms. We show how MCTS combines features of both RH and rollout to produce superior tracking performance while also conserving radar resources in a benchmark tracking simulation. 

A Cognitive Tracking Radar Using Continuous Space Monte Carlo Tree Search

This paper puts forward a cooperative node selection and resource allocation strategy for search and track applications in radar network. The strategy is proposed to minimize the total energy consumption for searching and multi-target tracking (MTT) tasks by optimizing radar node selection, dwell time and transmit power allocation of multiple radar nodes cooperatively, while satisfying the given search and track requirements and several system resource budgets. We utilize the detection probability as the metric for searching, and the predicted Bayesian Cramér-Rao lower bound (BCRLB) is derived as the criterion for MTT performance, which are both concerned as constraints in the optimization model. Subsequently, a two-step solution algorithm based on the interior point method and cyclic minimization framework is proposed to tackle the underlying optimization problem. Simulation results verify that the proposed strategy consumes the least energy while ensuring the specified search and track performance demands, when compared with other existing algorithms. 

Cooperative Node Selection and Resource Allocation for Search and Track Applications in Radar Network

We track moving targets with a distributed multiple-input multiple-output (MIMO) radar, for which the transmitters and receivers are appropriately paired and selected with a limited number of radar stations. We aim to maximize the sum of the signal-to-interference-plus-noise ratios (SINRs) of all the targets by sensibly selecting the transmitter-receiver pairs during the tracking period. A key is to model the optimization problem of selecting the transmitter-receiver pairs by a restless multi-armed bandit (RMAB) model that is able to formulate the time-varying signals of the transceiver channels whenever the channels are being probed or not. We regard the estimated mean reward (i.e., SINR) as the state of an arm. If an arm is probed, the estimated mean reward of the arm is the weighted sum of the observed reward and the predicted mean reward; otherwise, it is the predicted mean reward. We associate the predicted mean reward with the estimated mean reward at the previous time slot and the state of the target, which is estimated via the interacting multiple model-unscented Kalman filter (IMM-UKF). The optimized selection of transmitter-receiver pairs at each time is accomplished by using Binary Particle Swarm Optimization (BPSO) based on indexes of arms, each of which is designed by the upper confidence bound (UCB1) algorithm. Above all, a multi-group combinatorial-restless-bandit technique taking into account of different combinations of transmitters and receivers and the closed-loop scheme between transmitter-receiver pair selection and target state estimation, namely MG-CRB-CL, is developed to achieve a near-optimal selection strategy and improve multi-target tracking performance. Simulation results for different scenarios are provided to verify the effectiveness and superior performance of our MG-CRB-CL algorithm.

Combinatorial-restless-bandit-based Transmitter-Receiver Online Selection for Distributed MIMO Radars With Non-Stationary Channels

Frequency-agile (FA) transmission strategy plays a crucial role in radar antijamming applications. This strategy is usually designed or trained offline, which would lose the advantage of adaptability and flexibility when facing diverse jamming patterns and nonstationary target echoes. Considering the scenarios where a radar detects a target with strong nonstationary scattering characteristics under fast-variant interference, the radar is required to immediately adjust the FA transmission strategy to react to the variation of both jamming signals and target echoes quickly. In order to enhance radar transmission strategy in both generalization and flexibility aspects, an online FA strategy, called Combinatorial Discounted Thompson Sampling, is developed for the antijamming by exploiting the target's scattering change with the multiarmed bandit model. With the advantages of both optimal exploration and fast convergence, the proposed algorithm can efficiently adapt to the scattering fluctuation under dynamic frequency jamming emissions. Experimental comparisons with conventional deep reinforcement learning demonstrate the proposed algorithm's superiority for FA transmission strategy learning to boost radar detection performance under frequency response dynamics when avoiding frequency jamming.

Online Frequency-Agile Strategy for Radar Detection Based on Constrained Combinatorial Nonstationary Bandit

Artificial intelligence meets radar resource management: A comprehensive background and literature review

Active transmitter-receiver (TX-RX) subset selection facilitates efficient resource use and adaptation to varying target and propagation environments in distributed multiple-input multiple-output (MIMO) radar systems. The problem has been addressed in the literature and objective functions related to radar tasks that depend on the signal-to-interference-plus-noise ratio (SINR) have been proposed. The SINR values observed at the receivers can be estimated assuming a particular propagation environment and target models. In this paper, a novel machine learning approach is proposed in which no such assumptions are needed. We formulate the TX-RX subset selection as a multi-armed bandit (MAB) problem and further extend it to the combinatorial MAB framework. A variety of reinforcement learning algorithms developed for the MAB problem are employed to learn the optimal subset in real-time. It is shown that such algorithms can be effectively used for the TX-RX subset selection problem even in non-stationary scenarios.

/pdf/reinforcement-learning-based-transmitter-receiver-selection-3yix05gx6v.pdf

Reinforcement Learning Based Transmitter-Receiver Selection for Distributed MIMO Radars

An adaptive revisit interval selection (RIS) in multifunction radars is an integral part of efficient time budget management (TBM). In this paper, the RIS problem is formulated as a Markov decision process (MDP) with unknown state transition probabilities and reward distributions. A reward function is proposed to minimize the tracking load (TL) while maintaining the track loss probability (TLP) at a tolerable level. The reinforcement learning (RL) problem is solved using the Q-learning algorithm with an epsilon-greedy policy. Compared to a baseline algorithm, the RL approach was capable of maintaining the tracks while reducing the tracking load significantly.

/pdf/time-budget-management-in-multifunction-radars-using-2oxbbpx8s1.pdf

Petteri Pulkkinen

Papers

Reinforcement Learning Based Transmitter-Receiver Selection for Distributed MIMO Radars

Time Budget Management in Multifunction Radars Using Reinforcement Learning