Comparing simulation tools and experimental testbeds for wireless mesh networks

TLDR: This paper quantifies network behavioral differences between simulations and real-world testbed experiments and finds that in the PHY layer, the distribution of received signal strength in experiments is usually different from simulation due to the antenna diversity, and in the network layer, a few dominant routes exist in test bed experiments while routes in simulations are less stable.

Abstract: Wireless simulators provide full control to researchers in investigating traffic flow behavior, but do not always reflect real-world scenarios. Although previous work pointed out such shortages are due to the limitation of radio propagation models in the simulators, it is still unclear how these imperfect models affect network behavior and to what degree. In this paper, we quantify network behavioral differences between simulations and real-world testbed experiments. We compare and analyze the experimental results from indoor and outdoor experiments with the results from NS-2 and Qualnet simulations. We find that in the PHY layer, the distribution of received signal strength in experiments is usually different from simulation due to the antenna diversity. However, path loss, which is regarded as a dominating factor in simulator channel modeling, can be configured to match the real-world behavior. For the MAC layer, increasing traffic load on a flow may cause significant performance degradation in experiments, but it is not the case in simulations. Interference is inadequately captured in simulations and cannot show the flow level unfairness phenomenon. In the network layer, a few dominant routes exist in testbed experiments while routes in simulations are less stable. These findings give the wireless research community an improved overview about the differences between simulations and testbed experiments. They will help researchers to choose between simulations and experiments according to their particular research requirements.