The dynamic behavior of a data dissemination protocol for network programming at scale

TL;DR: It appears very hard to significantly improve upon the rate obtained by Deluge and it is argued that the rates obtained for dissemination are inherently lower than that for single path propagation.

Abstract: To support network programming, we present Deluge, a reliable data dissemination protocol for propagating large data objects from one or more source nodes to many other nodes over a multihop, wireless sensor network. Deluge builds from prior work in density-aware, epidemic maintenance protocols. Using both a real-world deployment and simulation, we show that Deluge can reliably disseminate data to all nodes and characterize its overall performance. On Mica2-dot nodes, Deluge can push nearly 90 bytes/second, one-ninth the maximum transmission rate of the radio supported under TinyOS. Control messages are limited to 18% of all transmissions. At scale, the protocol exposes interesting propagation dynamics only hinted at by previous dissemination work. A simple model is also derived which describes the limits of data propagation in wireless networks. Finally, we argue that the rates obtained for dissemination are inherently lower than that for single path propagation. It appears very hard to significantly improve upon the rate obtained by Deluge and we identify establishing a tight lower bound as an open problem.

Summary

1. INTRODUCTION

• Wireless sensor networks (WSNs) represent a new class of computing with large numbers of resource-constrained computing nodes cooperating on essentially a single application.
• These factors suggest that network programming (the programming of nodes by disseminating code over the network) is required for the success of WSNs.
• Third, the authors develop a simple model of Deluge’s propagation behavior and use it to identify different factors which limit the overall bandwidth of any multihop communication protocol.
• When nodes are not up-to-date, the broadcast rate is reduced, but is otherwise increased up to a specified limit.
• Section 7 discusses future directions, and this paper concludes with Section 8.

2. RELATED WORK

• The problem the authors address is an important special case of reliable data dissemination.
• The main differences include the dissemination of data over lossy local links, a constrained storage hierarchy, and the need to retain the most recent copy in order to disseminate data to additional nodes as they become connected over time.
• The epidemic property is important since WSNs experience high loss rates, asymmetric connectivity, and transient links due to node failures and repopulation.
• TinyOS [15] has included limited support for network programming via XNP [3].
• Deluge shares many ideas with MOAP, including the use of NACKs, unicast requests, broadcast data transmission, and windowing to efficiently manage which segments are required.

3. DELUGE

• Deluge is an epidemic protocol and operates as a statemachine where each node follows a set of strictly local rules to achieve a desired global behavior: the quick, reliable dis- semination of large data objects to many nodes.
• In its most basic form, each node occasionally advertises the most recent version of the data object it has available to whatever nodes that can hear its local broadcast.
• The first is its density-aware capability, where redundant advertisement and request messages are suppressed to minimize contention.
• It may continue making requests if sufficient progress is being made.
• Finally, Deluge emphasizes the use of spatial multiplexing to allow parallel transfers of data.

3.1 Data Representation

• To manage the large size, Deluge divides the data object into fixed-size pages.
• Instead, Deluge fragments the data object into P pages each of size Spage = N ·Spkt, where N is a fixed number of packets, as shown in Figure 1.
• Both packets and pages include 16-bit cyclic redundancy checks (CRCs).
• A complete description of the object is defined by the tuple (v,a), called the object profile, and is stored in non-volatile storage along with the data it represents.
• A node receiving an object profile for a newer version uses the age-vector to determine which pages need updating.

3.2 The Protocol

• A node operates in one of three states at any time: MAINTAIN, RX, or TX.
• A node in the MAINTAIN state is responsible for ensur- ing that all nodes within communication range have (i) the newest version of the object profile and (ii) all available data for the newest version.
• In Deluge, nodes request data from a single node S at a time.
• 2.2 Request A node in the RX state is responsible for actively re- questing the remaining packets required to complete page p = γ +.
• Nodes delay subsequent requests until they detect a period of silence equal to ω packet transmit times to help ensure the completion of data transmissions before requests are made.

3.3 Design Space

• The design space for data dissemination protocols is large and includes: methods for suppressing redundant control and data messages, selection of nodes to transmit data, use of forward error-correction (FEC), the fragmentation of data for spatial multiplexing, use of link quality estimates or other metrics to improve decisions, among others.
• Due to space limitations, the authors briefly mention some of their findings.
• The authors again tested Deluge by keeping only the request suppression and it also performed poorly, confirming the results presented by SPIN-RL’s authors that show it performing worse than naive flooding in lossy network models.
• The authors experimented with suppressing the transmission of data packets if k redundant data packets were overheard while in TX, where lower values of k represented more aggressive suppression.
• It was interesting to see that FEC improved performance in sparse networks while harming performance in dense networks.

4. EVALUATION METHODOLOGY

• The metrics the authors use to evaluate Deluge are driven by the primary motivation for this work: network programming.
• The authors list the metrics they consider, ordered from highest to lowest priority.
• In deployments, any interruption to their primary service caused by network programming should be minimized.
• To evaluate and investigate the behavior of Deluge, the authors use two separate mechanisms.
• The second is TOSSIM, a bit-level node simulator designed specifically for the TinyOS platform [7].

4.1 TinyOS Hardware

• To gather empirical data, the authors use a testbed composed Mica2-dots, a TinyOS supported hardware platform.
• The authors fully implemented Deluge in nesC and ran experiments using the Mica2-dot hardware platform with 75 nodes deployed non-uniformly in a 150’ by 100’ office environment [13].
• With the source placed at one corner, the diameter of the network is about five hops.
• To instrument a backchannel, the authors used specialized hardware to create a UART to TCP bridge, allowing nodes to transmit and receive messages over TCP through an Ethernet adapter.
• At the end of each experiment, the authors verify the integrity of the data object via the TCP connection.

4.2 TOSSIM

• In addition to hardware experiments, the authors use TOSSIM, a discrete-event network simulator, to investigate and evaluate Deluge at networks of much greater scale and differing structures.
• It is very useful for exposing overall behavior.
• Capturing sufficient detail when simulating the communication between nodes is essential since the behavior of dissemination protocols can be highly sensitive to low level factors.
• Each bit-error rate is independently chosen based on the distance between u and v and a random distribution derived from empirical data collected on Mica nodes.
• TOSSIM treats the transmission of each individual bit as an event.

5.1 Empirical Results

• The procedure of each experiment follows that of a normal deployment scenario.
• In its initial state, all but one of the nodes are deployed and operating in the steady-state, meaning that they have the same version v of the object profile and the same set of completed pages for version v.
• The narrow range of completion times is a direct result of spatial multiplexing, where the range of completion times represents the amount of time to flush the pipeline and is essentially the time to disseminate a single page.
• Again, these low numbers show the effectiveness of suppression on request transmissions.
• On average, a node receives about 3.35 times the minimum number of required data packets, while half receive fewer than 3.34 times the minimum number.

5.2 Simulation Results

• While the empirical results are promising, the authors were unable to experiment with networks of large scale since the testbed configuration does not scale easily.
• In the next section, the authors investigate the propagation dynamics of Deluge to see why density affects performance.
• Notice that the propagation speeds up as it approaches the edge of the network.
• The expected time for receiving advertisements from nodes closer to the source is given by E[TrAdv] = E[NtPkt] · τl 2 · (1 + E[NSupp]) (4) where E[NSupp] specifies the expected number of times that an advertisement from hop h is suppressed by hop h − 1 before transmitting an advertisement (in the linear case, E[NSupp] = 1).
• With a greater density, a larger delay period reduces any collisions caused by the hidden terminal problem.

6. CURRENT STATUS

• The fully implemented Deluge protocol is included as a core part of TinyOS 1.1.8 to support network programming on the Mica2, Mica2-dot, MicaZ, and Telos platforms, the latter two using 2.4 GHz, IEEE 802.15.4 radios.
• The current implementation consumes 84 bytes of RAM, of which 43 bytes is a message buffer.
• In addition to the features described in this paper, Deluge supports multiple objects on each node by including an object summary of each object in the advertisement.
• TOSBoot programs a node with the factory installed object on request by the user (through a radio command or a physical gesture) or if the node experiences failures.
• This factory installed object is crucial for node reliability by providing a solid software base that allows user interaction without programming the node using a physical connection.

7. LOOKING FORWARD

• While Deluge provides good, robust performance, there is room for potential improvement.
• The increased performance along the edge introduces an interesting concept for disseminating large data objects.
• In some cases, the authors were able to improve performance.
• Spatial multiplexing limits a node’s broadcast rate to no greater than onethird the maximum rate due to the single-channel, broadcast medium.
• While previous work has hinted at similar behaviors in real deployments, the effects are too complex to state anything conclusive and only small amounts of data are disseminated.

8. CONCLUSIONS

• The authors presented Deluge, a reliable data dissemination protocol for propagating large data objects from one or more source nodes to many other nodes over a multihop WSN.
• Control messages are limited to 18% of all transmissions.
• At scale, Deluge exposes propagation dynamics only hinted at by previous work, showing the impact of the hidden terminal problem on dissemination.
• Finally, the authors argued that dissemination is inherently slower than single path propagation and identified establishing a tight lower bound as an open problem.
• It appears very hard to significantly improve upon the rate obtained by Deluge.

Figures

Figure 8: Simulated Propagation Rate of a Single Page. Figure (a) shows the propagation rate through the diagonal and edge while Figures (b) and (c) compare the propagation rate for different values of τr.

Figure 10: Simulated Propagation for 1 Page from the Center in a 21× 21 Grid Topology.

Figure 3: TOSSIM Packet Loss Rates vs. Distance

Figure 7: Simulated Propagation Time for 5 Pages in a 20× 20 Grid Topology. Figures (a) - (d) and (e) - (h) are from a network with 15’ and 10’ node spacing, respectively.

Figure 1: Data Management Hierarchy.

Table 1: Contributions to Reduced Bandwidth from the Maximum Transmission Rate.

Figure 12: Comparison of Model with Simulation.

Figure 4: Empirical Data. These results are from a network of 75 nodes. Figure (a) shows the completion time when varying object size. Figure (b) shows the completion time for individual nodes. Figures (c)-(f) are derived from transmitting 20 pages over 75 nodes.

Figure 11: Simulated Propagation Time for Linear Structures. These results are from a 2 × N network where N = 4, 12, 20, . . . , 76 and 10’ spacing.

Figure 5: Simulated Propagation Time for Square Structures. These results are from a N × N network where N = 2, 4, 8, . . . , 20 with 15’ spacing. Figure (c) is for a 180′ × 180′ field for different densities.

Figure 6: Simulated Propagation Time for 1 Page in a 20× 20 Grid Topology. Figures (a) - (d) and (e) - (h) are from a network with 15’ and 10’ node spacing, respectively.

Figure 2: Pipelining. Example four-hop network. A three-hop spacing is required for effective spatial multiplexing. In this example, a simultaneous broadcast from A and C would collide at B.

Frequently asked questions

Q1. What are the contributions in "The dynamic behavior of a data dissemination protocol for network programming at scale" ?

To support network programming, the authors present Deluge, a reliable data dissemination protocol for propagating large data objects from one or more source nodes to many other nodes over a multihop, wireless sensor network. Using both a real-world deployment and simulation, the authors show that Deluge can reliably disseminate data to all nodes and characterize its overall performance. 

Q2. Why do both protocols emphasize hop-by-hop error recovery?

Because the cost of end-to-end repair is exponential with the path length, both protocols emphasize hop-by-hop error recovery where loss detection and recovery is limited to a small number of hops (ideally one). 

Q3. How do the authors measure the timing of the UART message?

By opening up sockets to each node from a desktop computer, the authors timestamp each UART message with precision on the order of milliseconds and track the propagation of each page. 

Q4. How much does spatial multiplexing limit a node’s broadcast rate?

spatial multiplexing limits a node’s broadcast rate to no greater than onethird the maximum rate due to the single-channel, broadcast medium. 

Q5. What is the way to reduce the propagation time?

One might suggest that starting the propagation in the centermight help to eliminate the behavior of following the edge and also decrease the propagation time by about half. 

Q6. What did the authors do with the suppression of data packets?

The authors experimented with suppressing the transmission of data packets if k redundant data packets were overheard while in TX, where lower values of k represented more aggressive suppression. 

Q7. What is the cause of the propagation of a single page along good links?

One cause is Deluge’s depth-first tendency, where propagation of a single page along good links is not blocked by delays caused by poor links. 

Q8. How does the propagation rate increase along the diagonal?

By doubling τr, the propagation rate along the diagonal improves by about 2.7 times while the propagation rate along the edge remains nearly identical, leading to an improvement in overall propagation performance. 

Q9. How does Deluge take advantage of the quick edges?

Even though placing the source at the center effectively reduces the the diameter by about half, Deluge is unable to take advantage of the quick edges since nodes in the center experience a greater number of collisions. 

Q10. What is the expected time to transmit just the data packets?

The expected time required to transmit just the data packets isE[Ttx] = E[NtPkt] · TtPkt ·N, (2)where TtPkt is the transmission time for a single packet. 

Q11. What is the only trigger that causes R to request data from S?

The only trigger that causes R to request data from S is the receipt of an advertisement stating the availability of a needed page. 

Q12. What is the effect of the SNACK constraint on the rate of requests from R?

the rate of requests from R will decrease with the decreasing advertisement rate in the steady state since S will not know that R is not up-to-date. 

Q13. Why is TOSSIM a good choice for simulating the network at the bit level?

The main advantage with simulating at the bit-level is that the transmission and reception of bits govern the actions of each layer, rather than modeling each layer with its own set of parameters. 

Q14. How many nodes did the authors use to evaluate and investigate the behavior of Deluge?

the authors used TOSSIM to evaluate and investigate the behavior of Deluge with network sizes on the order of hundreds of nodes and tens of hops. 

Q15. How many pages does Deluge take to disseminate?

For the linear case, the simulations show that Deluge takesabout 40 seconds to disseminate each page to 152 nodes across 15 hops. 

Q16. Why is it unlikely for many senders in a region to begin transmitting data?

Because requests are unicast to the node that most recently advertised, it is unlikely for many senders in a region to begin transmitting data. 

Q17. What is the behavior of the propagation of a square topology?

With this topology, the propagation behaves as expected: the propagation progresses at a fairly constant rate in a nice wavefront pattern from corner to corner. 

Q18. What is the way to improve the propagation time for a single page?

This structured approach should improve the propagation time for a single page, but inhibits the use of pipelining since it is more difficult to minimize interference between transfers of different pages. 

Q19. How do the authors plot the average advertisement rate for each node?

Using the transmission count information on advertisements, the authors plot a histogram of the average advertisement rate for each node by dividing the count by the completion time.