Showing papers on "Data flow diagram published in 2010"

FlumeJava: easy, efficient data-parallel pipelines

Abstract: MapReduce and similar systems significantly ease the task of writing data-parallel code. However, many real-world computations require a pipeline of MapReduces, and programming and managing such pipelines can be difficult. We present FlumeJava, a Java library that makes it easy to develop, test, and run efficient data-parallel pipelines. At the core of the FlumeJava library are a couple of classes that represent immutable parallel collections, each supporting a modest number of operations for processing them in parallel. Parallel collections and their operations present a simple, high-level, uniform abstraction over different data representations and execution strategies. To enable parallel operations to run efficiently, FlumeJava defers their evaluation, instead internally constructing an execution plan dataflow graph. When the final results of the parallel operations are eventually needed, FlumeJava first optimizes the execution plan, and then executes the optimized operations on appropriate underlying primitives (e.g., MapReduces). The combination of high-level abstractions for parallel data and computation, deferred evaluation and optimization, and efficient parallel primitives yields an easy-to-use system that approaches the efficiency of hand-optimized pipelines. FlumeJava is in active use by hundreds of pipeline developers within Google.

Processing data flows with a data flow processor

Abstract: An apparatus and method to distribute applications and services in and throughout a network and to secure the network includes the functionality of a switch with the ability to apply applications and services to received data according to respective subscriber profiles. Front-end processors, or Network Processor Modules (NPMs), receive and recognize data flows from subscribers, extract profile information for the respective subscribers, utilize flow scheduling techniques to forward the data to applications processors, or Flow Processor Modules (FPMs). The FPMs utilize resident applications to process data received from the NPMs. A Control Processor Module (CPM) facilitates applications processing and maintains connections to the NPMs, FPMs, local and remote storage devices, and a Management Server (MS) module that can monitor the health and maintenance of the various modules.

Nephele/PACTs: a programming model and execution framework for web-scale analytical processing

Abstract: We present a parallel data processor centered around a programming model of so called Parallelization Contracts (PACTs) and the scalable parallel execution engine Nephele [18]. The PACT programming model is a generalization of the well-known map/reduce programming model, extending it with further second-order functions, as well as with Output Contracts that give guarantees about the behavior of a function. We describe methods to transform a PACT program into a data flow for Nephele, which executes its sequential building blocks in parallel and deals with communication, synchronization and fault tolerance. Our definition of PACTs allows to apply several types of optimizations on the data flow during the transformation.The system as a whole is designed to be as generic as (and compatible to) map/reduce systems, while overcoming several of their major weaknesses: 1) The functions map and reduce alone are not sufficient to express many data processing tasks both naturally and efficiently. 2) Map/reduce ties a program to a single fixed execution strategy, which is robust but highly suboptimal for many tasks. 3) Map/reduce makes no assumptions about the behavior of the functions. Hence, it offers only very limited optimization opportunities. With a set of examples and experiments, we illustrate how our system is able to naturally represent and efficiently execute several tasks that do not fit the map/reduce model well.

Automatic Reverse Engineering of Data Structures from Binary Execution.

Abstract: With only the binary executable of a program, it is useful to discover the program's data structures and infer their syntactic and semantic definitions. Such knowledge is highly valuable in a variety of security and forensic applications. Although there exist efforts in program data structure inference, the existing solutions are not suitable for our targeted application scenarios. In this paper, we propose a reverse engineering technique to automatically reveal program data structures from binaries. Our technique, called REWARDS, is based on dynamic analysis. More specifically, each memory location accessed by the program is tagged with a timestamped type attribute. Following the program's runtime data flow, this attribute is propagated to other memory locations and registers that share the same type. During the propagation, a variable's type gets resolved if it is involved in a type-revealing execution point or type sink. More importantly, besides the forward type propagation, REWARDS involves a backward type resolution procedure where the types of some previously accessed variables get recursively resolved starting from a type sink. This procedure is constrained by the timestamps of relevant memory locations to disambiguate variables re-using the same memory location. In addition, REWARDS is able to reconstruct in-memory data structure layout based on the type information derived. We demonstrate that REWARDS provides unique benefits to two applications: memory image forensics and binary fuzzing for vulnerability discovery.

Automatic reverse engineering of data structures from binary execution

Abstract: With only the binary executable of a program, it is useful to discover the program's data structures and infer their syntactic and semantic definitions. Such knowledge is highly valuable in a variety of security and forensic applications. Although there exist efforts in program data structure inference, the existing solutions are not suitable for our targeted application scenarios. In this paper, we propose a reverse engineering technique to automatically reveal program data structures from binaries. Our technique, called REWARDS, is based on dynamic analysis. More specifically, each memory location accessed by the program is tagged with a timestamped type attribute. Following the program's runtime data flow, this attribute is propagated to other memory locations and registers that share the same type. During the propagation, a variable's type gets resolved if it is involved in a type-revealing execution point or type sink. More importantly, besides the forward type propagation, REWARDS involves a backward type resolution procedure where the types of some previously accessed variables get recursively resolved starting from a type sink. This procedure is constrained by the timestamps of relevant memory locations to disambiguate variables re-using the same memory location. In addition, REWARDS is able to reconstruct in-memory data structure layout based on the type information derived. We demonstrate that REWARDS provides unique benefits to two applications: memory image forensics and binary fuzzing for vulnerability discovery.

The X-shooter pipeline

Abstract: The X-shooter data reduction pipeline, as part of the ESO-VLT Data Flow System, provides recipes for Paranal Science Operations, and for Data Product and Quality Control Operations at Garching headquarters. At Paranal, it is used for the quick-look data evaluation. The pipeline recipes can be executed either with EsoRex at the command line level or through the Gasgano graphical user interface. The recipes are implemented with the ESO Common Pipeline Library (CPL). X-shooter is the first of the second generation of VLT instruments. It makes possible to collect in one shot the full spectrum of the target from 300 to 2500 nm, subdivided in three arms optimised for UVB, VIS and NIR ranges, with an efficiency between 15% and 35% including the telescope and the atmosphere, and a spectral resolution varying between 3000 and 17,000. It allows observations in stare, offset modes, using the slit or an IFU, and observing sequences nodding the target along the slit. Data reduction can be performed either with a classical approach, by determining the spectral format via 2D-polynomial transformations, or with the help of a dedicated instrument physical model to gain insight on the instrument and allowing a constrained solution that depends on a few parameters with a physical meaning. In the present paper we describe the steps of data reduction necessary to fully reduce science observations in the different modes with examples on typical data calibrations and observations sequences.

Finding low-utility data structures

Abstract: Many opportunities for easy, big-win, program optimizations are missed by compilers. This is especially true in highly layered Java applications. Often at the heart of these missed optimization opportunities lie computations that, with great expense, produce data values that have little impact on the program's final output. Constructing a new date formatter to format every date, or populating a large set full of expensively constructed structures only to check its size: these involve costs that are out of line with the benefits gained. This disparity between the formation costs and accrued benefits of data structures is at the heart of much runtime bloat.We introduce a run-time analysis to discover these low-utility data structures. The analysis employs dynamic thin slicing, which naturally associates costs with value flows rather than raw data flows. It constructs a model of the incremental, hop-to-hop, costs and benefits of each data structure. The analysis then identifies suspicious structures based on imbalances of its incremental costs and benefits. To decrease the memory requirements of slicing, we introduce abstract dynamic thin slicing, which performs thin slicing over bounded abstract domains. We have modified the IBM J9 commercial JVM to implement this approach.We demonstrate two client analyses: one that finds objects that are expensive to construct but are not necessary for the forward execution, and second that pinpoints ultimately-dead values. We have successfully applied them to large-scale and long-running Java applications. We show that these analyses are effective at detecting operations that have unbalanced costs and benefits.

RTL Hardware IP Protection Using Key-Based Control and Data Flow Obfuscation

Abstract: Recent trends of hardware intellectual property (IP) piracy and reverse engineering pose major business and security concerns to an IP-based system-on-chip (SoC) design flow. In this paper, we propose a Register Transfer Level (RTL) hardware IP protection technique based on low-overhead key-based obfuscation of control and data flow. The basic idea is to transform the RTL core into control and data flow graph (CDFG) and then integrate a well-obfuscated finite state machine (FSM) of special structure, referred as “Mode-Control FSM”, into the CDFG in a manner that normal functional behavior is enabled only after application of a specific input sequence. We provide formal analysis of the effectiveness of the proposed approach and present a simple metric to quantify the level of obfuscation. We also present an integrated design flow that implements the proposed obfuscation at low computational overhead. Simulation results for two open-source IP cores show that high levels of security is achievable at nominal area and power overheads under delay constraint.

Data flow mobility

Abstract: A wireless transmit/receive unit (WTRU) may communicate using a data flow that is defined according to flow identification information (FII). The WTRU may participate in the transfer of the data flow between access networks of diverse radio access technologies. The WTRU may communicate with a mobility function to obtain access network and mobility policy information. The mobility function may be, for example, an Access Network Discovery Function (ANDSF). The mobility policy information may describe the conditions by which the transfer of data flows between access networks may be permitted.

Anomaly detection in GPS data based on visual analytics

Abstract: Modern machine learning techniques provide robust approaches for data-driven modeling and critical information extraction, while human experts hold the advantage of possessing high-level intelligence and domain-specific expertise. We combine the power of the two for anomaly detection in GPS data by integrating them through a visualization and human-computer interaction interface. In this paper we introduce GPSvas (GPS Visual Analytics System), a system that detects anomalies in GPS data using the approach of visual analytics: a conditional random field (CRF) model is used as the machine learning component for anomaly detection in streaming GPS traces. A visualization component and an interactive user interface are built to visualize the data stream, display significant analysis results (i.e., anomalies or uncertain predications) and hidden information extracted by the anomaly detection model, which enable human experts to observe the real-time data behavior and gain insights into the data flow. Human experts further provide guidance to the machine learning model through the interaction tools; the learning model is then incrementally improved through an active learning procedure.

Database security via data flow processing

Abstract: An apparatus and method to distribute applications and services in and throughout a network and to secure the network includes the functionality of a switch with the ability to apply applications and services to received data according to respective subscriber profiles. Front-end processors, or Network Processor Modules (NPMs), receive and recognize data flows from subscribers, extract profile information for the respective subscribers, utilize flow scheduling techniques to forward the data to applications processors, or Flow Processor Modules (FPMs). The FPMs utilize resident applications to process data received from the NPMs. A Control Processor Module (CPM) facilitates applications processing and maintains connections to the NPMs, FPMs, local and remote storage devices, and a Management Server (MS) module that can monitor the health and maintenance of the various modules.

Securing a network with data flow processing

Abstract: An apparatus and method to distribute applications and services in and throughout a network and to secure the network includes the functionality of a switch with the ability to apply applications and services to received data according to respective subscriber profiles. Front-end processors, or Network Processor Modules (NPMs), receive and recognize data flows from subscribers, extract profile information for the respective subscribers, utilize flow scheduling techniques to forward the data to applications processors, or Flow Processor Modules (FPMs). The FPMs utilize resident applications to process data received from the NPMs. A Control Processor Module (CPM) facilitates applications processing and maintains connections to the NPMs, FPMs, local and remote storage devices, and a Management Server (MS) module that can monitor the health and maintenance of the various modules.

Method and system for guaranteed end-to-end data flows in a local networking domain

Abstract: A local manager in a local networking domain may configure, in conjunction with one or more switching devices, a plurality of network and/or switching devices to establish guaranteed end-to-end data flows in the local networking domain for servicing applications and/or processes running in the network devices. The network devices supporting or using guaranteed end-to-end data flows may determine data flow requirements for each serviced application, and may communicate the determined data flow requirements to switching devices supporting the local manager, for configuring the guaranteed end-to-end data flows. Data flow requirements may comprise bandwidth, quality of service (QoS), security, and/or service level agreement (SLA) related parameters. The network devices may allocate networking resources to guarantee the end-to-end data flow for each application running in each network device. Data flow routing tables maybe maintained and/or updated based on use of existing guaranteed end-to-end data flows.

Does the process map influence the outcome of quality improvement work? A comparison of a sequential flow diagram and a hierarchical task analysis diagram

Abstract: Many quality and safety improvement methods in healthcare rely on a complete and accurate map of the process. Process mapping in healthcare is often achieved using a sequential flow diagram, but there is little guidance available in the literature about the most effective type of process map to use. Moreover there is evidence that the organisation of information in an external representation affects reasoning and decision making. This exploratory study examined whether the type of process map - sequential or hierarchical - affects healthcare practitioners' judgments. A sequential and a hierarchical process map of a community-based anti coagulation clinic were produced based on data obtained from interviews, talk-throughs, attendance at a training session and examination of protocols and policies. Clinic practitioners were asked to specify the parts of the process that they judged to contain quality and safety concerns. The process maps were then shown to them in counter-balanced order and they were asked to circle on the diagrams the parts of the process where they had the greatest quality and safety concerns. A structured interview was then conducted, in which they were asked about various aspects of the diagrams. Quality and safety concerns cited by practitioners differed depending on whether they were or were not looking at a process map, and whether they were looking at a sequential diagram or a hierarchical diagram. More concerns were identified using the hierarchical diagram compared with the sequential diagram and more concerns were identified in relation to clinical work than administrative work. Participants' preference for the sequential or hierarchical diagram depended on the context in which they would be using it. The difficulties of determining the boundaries for the analysis and the granularity required were highlighted. The results indicated that the layout of a process map does influence perceptions of quality and safety problems in a process. In quality improvement work it is important to carefully consider the type of process map to be used and to consider using more than one map to ensure that different aspects of the process are captured.

An automated approach to transform use cases into activity diagrams

Abstract: Use cases are commonly used to structure and document requirements while UML activity diagrams are often used to visualize and formalize use cases, for example to support automated test case generation Therefore the automated support for the transition from use cases to activity diagrams would provide significant, practical help Additionally, traceability could be established through automated transformation, which could then be used for instance to relate requirements to design decisions and test cases In this paper, we propose an approach to automatically generate activity diagrams from use cases while establishing traceability links Data flow information can also be generated and added to these activity diagrams Our approach is implemented in a tool, which we used to perform five case studies The results show that high quality activity diagrams can be generated Our analysis also shows that our approach outperforms existing academic approaches and commercial tools

Wireless data communications employing IP flow mobility

Abstract: [0084] Techniques for IP flow mobility in a wireless data communications system, which allow for selective movement of IP data flows between different access networks supporting different access technology types. An access terminal (AT) is configured to selectively communicate using plural access technologies. Plural IP data flows are established between the AT and a first access network using a first technology type. A flow mobility trigger condition is detected, and in response, at least one IP data flow is moved to a second access network using a second technology, while maintaining at least another of the data flows to the first access network, and using the same IP address for the AT for data flows handled by both access networks, by using a mapping function between packet- filters for data flows and the access technology. The techniques may be used in conjunction with IP mobility protocols such as PMIP and CMIP.

Meter-based hierarchical bandwidth sharing

Abstract: Example methods and apparatus for hierarchical bandwidth management are disclosed An example method includes, receiving a data packet included in a first data traffic flow having a first rate of traffic The example method further includes marking the data packet with a first marker type if the first rate of traffic is less than or equal to a first threshold, otherwise marking the data packet with a second marker type The example method also includes combining the first data traffic flow with a second data traffic flow having a second rate of traffic to produce a third data traffic flow having a third rate of traffic The example method still further includes, if the data packet is marked with the first marker type, forwarding the data packet in the third data flow The example method yet further includes, if the data packet is marked with the second marker type and the third rate of traffic is less than or equal to a second threshold, forwarding the data packet in the third data flow, otherwise, discarding the packet

Embedded system development

Abstract: A computer-implemented method of automatically generating an embedded system on the basis of an original computer program, comprising analyzing the original computer program, comprising a step of compiling the original computer program into an executable to obtain data flow graphs with static data dependencies and a step of executing the executable using test data to provide dynamic data dependencies as communication patterns between load and store operations of the original computer program, and a step of transforming the original computer program into an intermediary computer program that exhibits multi-threaded parallelism with inter-thread communication, which comprises identifying at least one static and/or dynamic data dependency that crosses a thread boundary and converting said data dependency into a buffered communication channel with read/write access.

A VLSI Architecture and Algorithm for Lucas–Kanade-Based Optical Flow Computation

Abstract: Optical flow computation in vision-based systems demands substantial computational power and storage area. Hence, to enable real-time processing at high resolution, the design of application-specific system for optic flow becomes essential. In this paper, we propose an efficient VLSI architecture for the accurate computation of the Lucas-Kanade (L-K)-based optical flow. The L-K algorithm is first converted to a scaled fixed-point version, with optimal bit widths, for improving the feasibility of high-speed hardware implementation without much loss in accuracy. The algorithm is mapped onto an efficient VLSI architecture and the data flow exploits the principles of pipelining and parallelism. The optical flow estimation involves several tasks such as Gaussian smoothing, gradient computation, least square matrix calculation, and velocity estimation, which are processed in a pipelined fashion. The proposed architecture was simulated and verified by synthesizing onto a Xilinx Field Programmable Gate Array, which utilize less than 40% of system resources while operating at a frequency of 55 MHz. Experimental results on benchmark sequences indicate 42% improvement in accuracy and a speed up of five times, compared to a recent hardware implementation of the L-K algorithm.

ParSym: Parallel symbolic execution

Abstract: Scaling software analysis techniques based on source-code, such as symbolic execution and data flow analyses, remains a challenging problem for systematically checking software systems. The increasing availability of clusters of commodity machines provides novel opportunities to scale these techniques using parallel algorithms. This paper presents ParSym, a novel parallel algorithm for scaling symbolic execution using a parallel implementation. In every iteration ParSym explores multiple branches of a path condition in parallel by distributing them among available workers resulting in an efficient parallel version of symbolic execution. Experimental results show that symbolic execution is highly scalable using parallel algorithms: using 512 processors, more than two orders of magnitude speedup are observed.

A profile-based tool for finding pipeline parallelism in sequential programs

Abstract: Traditional static analysis fails to auto-parallelize programs with a complex control and data flow. Furthermore, thread-level parallelism in such programs is often restricted to pipeline parallelism, which can be hard to discover by a programmer. In this paper we propose a tool that, based on profiling information, helps the programmer to discover parallelism. The programmer hand-picks the code transformations from among the proposed candidates which are then applied by automatic code transformation techniques. This paper contributes to the literature by presenting a profiling tool for discovering thread-level parallelism. We track dependencies at the whole-data structure level rather than at the element level or byte level in order to limit the profiling overhead. We perform a thorough analysis of the needs and costs of this technique. Furthermore, we present and validate the belief that programs with complex control and data flow contain significant amounts of exploitable coarse-grain pipeline parallelism in the program's outer loops. This observation validates our approach to whole-data structure dependencies. As state-of-the-art compilers focus on loops iterating over data structure members, this observation also explains why our approach finds coarse-grain pipeline parallelism in cases that have remained out of reach for state-of-the-art compilers. In cases where traditional compilation techniques do find parallelism, our approach allows to discover higher degrees of parallelism, allowing a 40% speedup over traditional compilation techniques. Moreover, we demonstrate real speedups on multiple hardware platforms.

Characterizing data flow in a network based on a footprint measurement and taking action to avoid packet loss including buffer scaling or traffic shaping by adapting the footprint measurement

Abstract: A method for characterising a data flow to be transferred over a network path of a network, whereby the network path has at least one network device susceptible of network congestion. The method includes the step of determining a footprint measure of the data flow. The footprint measure is indicative of a possible difference between the total amount of incoming data and the total amount of outgoing data in the network device over a time interval having a duration of one or more time units, whereby that time unit is so chosen that individual data units of the data flow are distinguishable at byte level by the network device. The invention also relates to a device for performing the method.

ConnectX-2 InfiniBand Management Queues: First Investigation of the New Support for Network Offloaded Collective Operations

Abstract: This paper introduces the newly developed Infini- Band (IB) Management Queue capability, used by the Host Channel Adapter (HCA) to manage network task data flow dependancies, and progress the communications associated with such flows. These tasks include sends, receives, and the newly supported wait task, and are scheduled by the HCA based on a data dependency description provided by the user. This functionality is supported by the ConnectX-2 HCA, and provides the means for delegating collective communication management and progress to the HCA, also known as collective communication offload. This provides a means for overlapping collective communications managed by the HCA and computation on the Central Processing Unit (CPU), thus making it possible to reduce the impact of system noise on parallel applications using collective operations. This paper further describes how this new capability can be used to implement scalable Message Passing Interface (MPI) collective operations, describing the high level details of how this new capability is used to implement the MPI Barrier collective operation, focusing on the latency sensitive performance aspects of this new capability. This paper concludes with small scale bench- mark experiments comparing implementations of the barrier collective operation, using the new network offload capabilities, with established point-to-point based implementations of these same algorithms, which manage the data flow using the central processing unit. These early results demonstrate the promise this new capability provides to improve the scalability of high- performance applications using collective communications. The latency of the HCA based implementation of the barrier is similar to that of the best performing point-to-point based implementation managed by the central processing unit, starting to outperform these as the number of processes involved in the collective operation increases.

Threat Risk Modeling

Abstract: Application development security utilizes a list of threats that identify and organize application security classes of attack. The developed system is decomposed into relevant components and then each component is analyzed for threats. Data flow diagrams (DFDs) are typically used to graphically represent a system. In this paper we propose a flow-based model as an alternative methodology for identification and classification of threats.

Formalization of the data flow diagram rules for consistency check

Abstract: In system development life cycle (SDLC), a system model can be developed using Data Flow Diagram (DFD). DFD is graphical diagrams for specifying, constructing and visualizing the model of a system. DFD is used in defining the requirements in a graphical view. In this paper, we focus on DFD and its rules for drawing and defining the diagrams. We then formalize these rules and develop the tool based on the formalized rules. The formalized rules for consistency check between the diagrams are used in developing the tool. This is to ensure the syntax for drawing the diagrams is correct and strictly followed. The tool automates the process of manual consistency check between data flow diagrams.

On Detecting Data Flow Errors in Workflows

Abstract: When designing a business workflow, it is customary practice to create the control flow structure first and to ensure its correctness. Information about the flow of data is introduced subsequently into the workflow and its correctness is independently verified. Improper specification of data requirements of tasks and XOR splits can cause problems such as wrong branching at XOR splits and the failure of tasks to execute. Here we present a graph traversal algorithm called GTforDF for detecting data flow errors in both nested and unstructured workflows, and illustrate its operation on realistic examples. Two of these have interconnected loops and are free of control flow errors, and the third one is an unstructured loop-free workflow. Our approach extends and generalizes data flow verification methods that have been recently proposed. It also makes use of the concept of corresponding pairs lately introduced in control flow verification. It thus has the potential for development into a unified algorithmic procedure for the concurrent detection of control flow and data flow errors. The correctness of the algorithm has been proved theoretically. It has also been tested experimentally on many examples.

Interface device and method for command processing

Abstract: Embodiments of the present invention provide an interface device and method for command processing for commands requiring data flow in both directions on a Fiber Channel or other data transport protocol exchange. The commands can include proprietary commands, SCSI linked commands or other commands known in the art. According to one embodiment, and interface device can assign a command a data flow direction indicator. When a reply to the command is received, the interface device can determine if the reply is expected or unexpected based on the data flow direction specified by the data flow direction indicator. If the reply is unexpected, the interface device can determine whether to process the reply. According to one embodiment, the data flow direction indicator can be the exchange identification.

StarFlow: A Script-Centric Data Analysis Environment

Abstract: We introduce StarFlow, a script-centric environment for data analysis. StarFlow has four main features: (1) extraction of control and data-flow dependencies through a novel combination of static analysis, dynamic runtime analysis, and user annotations, (2) command-line tools for exploring and propagating changes through the resulting dependency network, (3) support for workflow abstractions enabling robust parallel executions of complex analysis pipelines, and (4) a seamless interface with the Python scripting language. We describe real applications of StarFlow, including automatic parallelization of complex workflows in the cloud.