Showing papers on "Benchmark (computing) published in 2014"

CDnet 2014: An Expanded Change Detection Benchmark Dataset

Abstract: Change detection is one of the most important lowlevel tasks in video analytics. In 2012, we introduced the changedetection.net (CDnet) benchmark, a video dataset devoted to the evalaution of change and motion detection approaches. Here, we present the latest release of the CDnet dataset, which includes 22 additional videos (70; 000 pixel-wise annotated frames) spanning 5 new categories that incorporate challenges encountered in many surveillance settings. We describe these categories in detail and provide an overview of the results of more than a dozen methods submitted to the IEEE Change DetectionWorkshop 2014. We highlight strengths and weaknesses of these methods and identify remaining issues in change detection.

Creating Summaries from User Videos

Abstract: This paper proposes a novel approach and a new benchmark for video summarization. Thereby we focus on user videos, which are raw videos containing a set of interesting events. Our method starts by segmenting the video by using a novel “superframe” segmentation, tailored to raw videos. Then, we estimate visual interestingness per superframe using a set of low-, mid- and high-level features. Based on this scoring, we select an optimal subset of superframes to create an informative and interesting summary. The introduced benchmark comes with multiple human created summaries, which were acquired in a controlled psychological experiment. This data paves the way to evaluate summarization methods objectively and to get new insights in video summarization. When evaluating our method, we find that it generates high-quality results, comparable to manual, human-created summaries.

Real-Time Implementation of Fault-Tolerant Control Systems With Performance Optimization

Abstract: In this paper, two online schemes for an integrated design of fault-tolerant control (FTC) systems with application to Tennessee Eastman (TE) benchmark are proposed. Based on the data-driven design of the proposed fault-tolerant architecture whose core is an observer/residual generator based realization of the Youla parameterization of all stabilization controllers, FTC is achieved by an adaptive residual generator for the online identification of the fault diagnosis relevant vectors, and an iterative optimization method for system performance enhancement. The performance and effectiveness of the proposed schemes are demonstrated through the TE benchmark model.

Single-Image Super-Resolution: A Benchmark

Abstract: Single-image super-resolution is of great importance for vision applications, and numerous algorithms have been proposed in recent years. Despite the demonstrated success, these results are often generated based on different assumptions using different datasets and metrics. In this paper, we present a systematic benchmark evaluation for state-of-the-art single-image super-resolution algorithms. In addition to quantitative evaluations based on conventional full-reference metrics, human subject studies are carried out to evaluate image quality based on visual perception. The benchmark evaluations demonstrate the performance and limitations of state-of-the-art algorithms which sheds light on future research in single-image super-resolution.

Binary bat algorithm

Abstract: Bat algorithm (BA) is one of the recently proposed heuristic algorithms imitating the echolocation behavior of bats to perform global optimization. The superior performance of this algorithm has been proven among the other most well-known algorithms such as genetic algorithm (GA) and particle swarm optimization (PSO). However, the original version of this algorithm is suitable for continuous problems, so it cannot be applied to binary problems directly. In this paper, a binary version of this algorithm is proposed. A comparative study with binary PSO and GA over twenty-two benchmark functions is conducted to draw a conclusion. Furthermore, Wilcoxon's rank-sum nonparametric statistical test was carried out at 5 % significance level to judge whether the results of the proposed algorithm differ from those of the other algorithms in a statistically significant way. The results prove that the proposed binary bat algorithm (BBA) is able to significantly outperform others on majority of the benchmark functions. In addition, there is a real application of the proposed method in optical engineering called optical buffer design at the end of the paper. The results of the real application also evidence the superior performance of BBA in practice.

A benchmark for comparison of cell tracking algorithms

Abstract: Motivation: Automatic tracking of cells in multidimensional time� lapse fluorescence microscopy is an important task in many biomed� ical applications. A novel framework for objective evaluation of cell tracking algorithms has been established under the auspices of the IEEE International Symposium on Biomedical Imaging 2013 Cell Tracking Challenge. In this paper, we present the logistics, datasets, methods and results of the challenge and lay down the principles for future uses of this benchmark. Results: The main contributions of the challenge include the crea� tion of a comprehensive video dataset repository and the definition of objective measures for comparison and ranking of the algorithms. With this benchmark, six algorithms covering a variety of segmenta� tion and tracking paradigms have been compared and ranked based on their performance on both synthetic and real datasets. Given the diversity of the datasets, we do not declare a single winner of the challenge. Instead, we present and discuss the results for each individual dataset separately. Availability and implementation: The challenge website (http://www.codesolorzano.com/celltrackingchallenge) provides access to the training and competition datasets, along with the ground truth of the training videos. It also provides access to Win� dows and Linux executable files of the evaluation software and most of the algorithms that competed in the challenge.

One billion word benchmark for measuring progress in statistical language modeling.

Abstract: We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned Kneser-Ney 5-gram model achieves perplexity 67.6; a combination of techniques leads to 35% reduction in perplexity, or 10% reduction in cross-entropy (bits), over that baseline. The benchmark is available as a this http URL project; besides the scripts needed to rebuild the training/held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models.

An Empirical Investigation of V-I Trajectory Based Load Signatures for Non-Intrusive Load Monitoring

Abstract: Choice of load signature or feature space is one of the most fundamental design choices for non-intrusive load monitoring or energy disaggregation problem. Electrical power quantities, harmonic load characteristics, canonical transient and steady-state waveforms are some of the typical choices of load signature or load signature basis for current research addressing appliance classification and prediction. This paper expands and evaluates appliance load signatures based on V-I trajectory-the mutual locus of instantaneous voltage and current waveforms-for precision and robustness of prediction in classification algorithms used to disaggregate residential overall energy use and predict constituent appliance profiles. We also demonstrate the use of variants of differential evolution as a novel strategy for selection of optimal load models in context of energy disaggregation. A publicly available benchmark dataset REDD is employed for evaluation purposes. Our experimental evaluations indicate that these load signatures, in conjunction with a number of popular classification algorithms, offer better or generally comparable overall precision of prediction, robustness and reliability against dynamic, noisy and highly similar load signatures with reference to electrical power quantities and harmonic content. Herein, wave-shape features are found to be an effective new basis of classification and prediction for semi-automated energy disaggregation and monitoring.

A Base Integer Programming Model and Benchmark Suite for Liner-Shipping Network Design

Abstract: The liner-shipping network design problem is to create a set of nonsimple cyclic sailing routes for a designated fleet of container vessels that jointly transports multiple commodities. The objective is to maximize the revenue of cargo transport while minimizing the costs of operation. The potential for making cost-effective and energy-efficient liner-shipping networks using operations research OR is huge and neglected. The implementation of logistic planning tools based upon OR has enhanced performance of airlines, railways, and general transportation companies, but within the field of liner shipping, applications of OR are scarce. We believe that access to domain knowledge and data is a barrier for researchers to approach the important liner-shipping network design problem. The purpose of the benchmark suite and the paper at hand is to provide easy access to the domain and the data sources of liner shipping for OR researchers in general. We describe and analyze the liner-shipping domain applied to network design and present a rich integer programming model based on services that constitute the fixed schedule of a liner shipping company. We prove the liner-shipping network design problem to be strongly NP-hard. A benchmark suite of data instances to reflect the business structure of a global liner shipping network is presented. The design of the benchmark suite is discussed in relation to industry standards, business rules, and mathematical programming. The data are based on real-life data from the largest global liner-shipping company, Maersk Line, and supplemented by data from several industry and public stakeholders. Computational results yielding the first best known solutions for six of the seven benchmark instances is provided using a heuristic combining tabu search and heuristic column generation.

Accurate Scene Text Recognition Based on Recurrent Neural Network

Abstract: Scene text recognition is a useful but very challenging task due to uncontrolled condition of text in natural scenes. This paper presents a novel approach to recognize text in scene images. In the proposed technique, a word image is first converted into a sequential column vectors based on Histogram of Oriented Gradient (HOG). The Recurrent Neural Network (RNN) is then adapted to classify the sequential feature vectors into the corresponding word. Compared with most of the existing methods that follow a bottom-up approach to form words by grouping the recognized characters, our proposed method is able to recognize the whole word images without character-level segmentation and recognition. Experiments on a number of publicly available datasets show that the proposed method outperforms the state-of-the-art techniques significantly. In addition, the recognition results on publicly available datasets provide a good benchmark for the future research in this area.

Towards a Big Data Curated Benchmark of Inter-project Code Clones

Abstract: Recently, new applications of code clone detection and search have emerged that rely upon clones detected across thousands of software systems. Big data clone detection and search algorithms have been proposed as an embedded part of these new applications. However, there exists no previous benchmark data for evaluating the recall and precision of these emerging techniques. In this paper, we present a big data clone detection benchmark that consists of known true and false positive clones in a big data inter-project Java repository. The benchmark was built by mining and then manually checking clones of ten common functionalities. The benchmark contains six million true positive clones of different clone types: Type-1, Type-2, Type-3 and Type-4, including various strengths of Type-3 similarity (strong, moderate, weak). These clones were found by three judges over 216 hours of manual validation efforts. We show how the benchmark can be used to measure the recall and precision of clone detection techniques.

Starexec: A cross-community infrastructure for logic solving

Abstract: We introduce StarExec, a public web-based service built to facilitate the experimental evaluation of logic solvers, broadly understood as automated tools based on formal reasoning. Examples of such tools include theorem provers, SAT and SMT solvers, constraint solvers, model checkers, and software verifiers. The service, running on a compute cluster with 380 processors and 23 terabytes of disk space, is designed to provide a single piece of storage and computing infrastructure to logic solving communities and their members. It aims at reducing duplication of effort and resources as well as enabling individual researchers or groups with no access to comparable infrastructure. StarExec allows community organizers to store, manage and make available benchmark libraries; competition organizers to run logic solver competitions; and community members to do comparative evaluations of logic solvers on public or private benchmark problems.

Convolutional Neural Network-based Place Recognition

Abstract: Recently Convolutional Neural Networks (CNNs) have been shown to achieve state-of-the-art performance on various classification tasks In this paper, we present for the first time a place recognition technique based on CNN models, by combining the powerful features learnt by CNNs with a spatial and sequential filter Applying the system to a 70 km benchmark place recognition dataset we achieve a 75% increase in recall at 100% precision, significantly outperforming all previous state of the art techniques We also conduct a comprehensive performance comparison of the utility of features from all 21 layers for place recognition, both for the benchmark dataset and for a second dataset with more significant viewpoint changes

Powertrain control verification benchmark

Abstract: Industrial control systems are often hybrid systems that are required to satisfy strict performance requirements. Verifying designs against requirements is a difficult task, and there is a lack of suitable open benchmark models to assess, evaluate, and compare tools and techniques. Benchmark models can be valuable for the hybrid systems research community, as they can communicate the nature and complexity of the problems facing industrial practitioners. We present a collection of benchmark problems from the automotive powertrain control domain that are focused on verification for hybrid systems; the problems are intended to challenge the research community while maintaining a manageable scale. We present three models of a fuel control system, each with a unique level of complexity, along with representative requirements in signal temporal logic (STL). We provide results obtained by applying a state of the art analysis tool to these models, and finally, we discuss challenge problems for the research community.

Strengthening the Effectiveness of Pedestrian Detection with Spatially Pooled Features

Abstract: We propose a simple yet effective approach to the problem of pedestrian detection which outperforms the current state-of-the-art. Our new features are built on the basis of low-level visual features and spatial pooling. Incorporating spatial pooling improves the translational invariance and thus the robustness of the detection process. We then directly optimise the partial area under the ROC curve (pAUC) measure, which concentrates detection performance in the range of most practical importance. The combination of these factors leads to a pedestrian detector which outperforms all competitors on all of the standard benchmark datasets. We advance state-of-the-art results by lowering the average miss rate from 13% to 11% on the INRIA benchmark, 41% to 37% on the ETH benchmark, 51% to 42% on the TUD-Brussels benchmark and 36% to 29% on the Caltech-USA benchmark.

General-purpose code acceleration with limited-precision analog computation

Abstract: As improvements in per-transistor speed and energy efficiency diminish, radical departures from conventional approaches are becoming critical to improving the performance and energy efficiency of general-purpose processors. We propose a solution--from circuit to compiler-that enables general-purpose use of limited-precision, analog hardwareto accelerate "approximable" code---code that can tolerate imprecise execution. We utilize an algorithmic transformation that automatically converts approximable regions of code from a von Neumann model to an "analog" neural model. We outline the challenges of taking an analog approach, including restricted-range value encoding, limited precision in computation, circuit inaccuracies, noise, and constraints on supported topologies. We address these limitations with a combination of circuit techniques, a hardware/software interface, neuralnetwork training techniques, and compiler support. Analog neural acceleration provides whole application speedup of 3.7x and energy savings of 6.3x with quality loss less than 10% for all except one benchmark. These results show that using limited-precision analog circuits for code acceleration, through a neural approach, is both feasible and beneficial over a range of approximation-tolerant, emerging applications including financial analysis, signal processing, robotics, 3D gaming, compression, and image processing

A benchmark study of large-scale unconstrained face recognition

Abstract: Many efforts have been made in recent years to tackle the unconstrained face recognition challenge. For the benchmark of this challenge, the Labeled Faces in the Wild (LFW) database has been widely used. However, the standard LFW protocol is very limited, with only 3,000 genuine and 3,000 impostor matches for classification. Today a 97% accuracy can be achieved with this benchmark, remaining a very limited room for algorithm development. However, we argue that this accuracy may be too optimistic because the underlying false accept rate may still be high (e.g. 3%). Furthermore, performance evaluation at low FARs is not statistically sound by the standard protocol due to the limited number of impostor matches. Thereby we develop a new benchmark protocol to fully exploit all the 13,233 LFW face images for large-scale unconstrained face recognition evaluation under both verification and open-set identification scenarios, with a focus at low FARs. Based on the new benchmark, we evaluate 21 face recognition approaches by combining 3 kinds of features and 7 learning algorithms. The benchmark results show that the best algorithm achieves 41.66% verification rates at FAR=0.1%, and 18.07% open-set identification rates at rank 1 and FAR=1%. Accordingly we conclude that the large-scale unconstrained face recognition problem is still largely unresolved, thus further attention and effort is needed in developing effective feature representations and learning algorithms. We thereby release a benchmark tool to advance research in this field.

A Thermal Infrared Video Benchmark for Visual Analysis

Abstract: We hereby publish a new thermal infrared video benchmark, called TIV, for various visual analysis tasks, which include single object tracking in clutter, multi-object tracking in single or multiple views, analyzing motion patterns of large groups, and censusing wild animals in flight. Our data describe real world scenarios, such as bats emerging from their caves in large numbers, a crowded street view during a marathon competition, and students walking through an atrium during class break. We also introduce baseline methods and evaluation protocols for these tasks. Our TIV benchmark enriches and diversifies video data sets available to the research community with thermal infrared footage, which poses new and challenging video analysis problems. We hope the TIV benchmark will help the community to better understand these interesting problems, generate new ideas, and value it as a testbed to compare solutions.

Multi-core Interference-Sensitive WCET Analysis Leveraging Runtime Resource Capacity Enforcement

Abstract: The performance and power efficiency of multi-core processors are attractive features for safety-critical applications, as in avionics But increased integration and average-case performance optimisations pose challenges when deploying them for such domains In this paper we propose a novel approach to compute an is WCET considering variable access delays due to the concurrent use of shared resources in multi-core processors, particularly focusing on shared interconnects and main memory Thereby we tackle the problem of temporal partitioning as required by safety-critical applications In particular, we introduce additional phases to state-of-the-art timing analysis techniques to analyse an application's resource usage and compute an interference delay We further complement the offline analysis with a runtime monitoring concept to enforce resource usage guarantees The concepts are evaluated on Free scale's P4080 multi-core processor in combination with SYSGO's commercial real-time operating system Pike OS and Abs Int's timing analysis framework aiT We abstract real applications' behaviour using a representative task set of the EEMBC Auto bench benchmark suite Our results show a reduction of up to 53% of the multi-core WCET, while implementing full transparency to the temporal and functional behaviour of applications, enabling the seamless integration of legacy applications

Use of the stair vision library within the ISPRS 2D semantic labeling benchmark (Vaihingen)

Abstract: This report describes experiments conducted using the multi-class image classification framework implemented in the stair vision library (SVL, (Gould et al., 2008)) in the context of the ISPRS 2D semantic labeling benchmark. The motivation was to get results from a well-established and public available software (Gould, 2014), as a kind of baseline. Besides the use of features implemented in the SVL which makes use of three channel images, assuming RGB, we also included features derived from the height model and the NDVI which is specific here, because the benchmark dataset provides surface models and CIR images. Another point of interest concerned the impact the segmentation had on the overall result. To this end a pre-study was performed where different parameters for the graph-based segmentation method introduced by Felzenszwalb and Huttelocher (2004) have been tested, in addition we only applied a simple chessboard segmentation. Other experiments focused on the question whether the conditional random field classification approach helps to enhance the overall performance. The official evaluation of all experiments described here is available at http://www2.isprs.org/vaihingen-2d-semantic-labeling-contest.html (SVL_1 to SVL_6). The normalized height models are available through the ReseachGate profile of the author (http://www.researchgate.net/profile/Markus_Gerke)

A novel fruit fly optimization algorithm for the semiconductor final testing scheduling problem

Abstract: In this paper, a novel fruit fly optimization algorithm (nFOA) is proposed to solve the semiconductor final testing scheduling problem (SFTSP). First, a new encoding scheme is presented to represent solutions reasonably, and a new decoding scheme is presented to map solutions to feasible schedules. Second, it uses multiple fruit fly groups during the evolution process to enhance the parallel search ability of the FOA. According to the characteristics of the SFTSP, a smell-based search operator and a vision-based search operator are well designed for the groups to stress exploitation. Third, to simulate the information communication behavior among fruit flies, a cooperative search process is developed to stress exploration. The cooperative search process includes a modified improved precedence operation crossover (IPOX) and a modified multipoint preservative crossover (MPX) based on two popular structures of the flexible job shop scheduling. Moreover, the influence of the parameter setting is investigated by using Taguchi method of design-of-experiment (DOE), and suitable values are determined for key parameters. Finally, computational tests results with some benchmark instances and the comparisons to some existing algorithms are provided, which demonstrate the effectiveness and the efficiency of the nFOA in solving the SFTSP.

Range-Sample Depth Feature for Action Recognition

Abstract: We propose binary range-sample feature in depth. It is based on τ tests and achieves reasonable invariance with respect to possible change in scale, viewpoint, and background. It is robust to occlusion and data corruption as well. The descriptor works in a high speed thanks to its binary property. Working together with standard learning algorithms, the proposed descriptor achieves state-of-theart results on benchmark datasets in our experiments. Impressively short running time is also yielded.

Chisel: reliability- and accuracy-aware optimization of approximate computational kernels

Abstract: The accuracy of an approximate computation is the distance between the result that the computation produces and the corresponding fully accurate result. The reliability of the computation is the probability that it will produce an acceptably accurate result. Emerging approximate hardware platforms provide approximate operations that, in return for reduced energy consumption and/or increased performance, exhibit reduced reliability and/or accuracy. We present Chisel, a system for reliability- and accuracy-aware optimization of approximate computational kernels that run on approximate hardware platforms. Given a combined reliability and/or accuracy specification, Chisel automatically selects approximate kernel operations to synthesize an approximate computation that minimizes energy consumption while satisfying its reliability and accuracy specification. We evaluate Chisel on five applications from the image processing, scientific computing, and financial analysis domains. The experimental results show that our implemented optimization algorithm enables Chisel to optimize our set of benchmark kernels to obtain energy savings from 8.7% to 19.8% compared to the fully reliable kernel implementations while preserving important reliability guarantees.

Chisel: Reliability- and Accuracy-Aware Optimization of Approximate Computational Kernels

Abstract: The accuracy of an approximate computation is the distance between the result that the computation produces and the corresponding fully accurate result. The reliability of the computation is the probability that it will produce an acceptably accurate result. Emerging approximate hardware platforms provide approximate operations that, in return for reduced energy consumption and/or increased performance, exhibit reduced reliability and/or accuracy. We present Chisel, a system for reliability- and accuracy-aware optimization of approximate computational kernels that run on approximate hardware platforms. Given a combined reliability and/or accuracy specification, Chisel automatically selects approximate kernel operations to synthesize an approximate computation that minimizes energy consumption while satisfying its reliability and accuracy specification. We evaluate Chisel on five applications from the image processing, scientific computing, and financial analysis domains. The experimental results show that our implemented optimization algorithm enables Chisel to optimize our set of benchmark kernels to obtain energy savings from 8.7% to 19.8% compared to the fully reliable kernel implementations while preserving important reliability guarantees.

S2CBench : synthesizable systemc benchmark suite for high-level synthesis

Abstract: High-level synthesis (HLS) is being increasingly used for commercial VLSI designs. This has led to the proliferation of many HLS tools. In order to evaluate their performance and functionalities, a standard benchmark suite in a common language supported by all of them is required. This letter presents a benchmark suite, which complies with the latest Synthesizable SystemC standard, called S2CBench. The benchmarks have been carefully chosen to not only include applications of different sizes and from various domains typically used in HLS (e.g., encryption, image and DSP application), but also to test specific optimization techniques in each of them. This allows an easy comparison of not only quality of results (QoR) of the different HLS tools under review, but also to test their completeness.

SMC structural health monitoring benchmark problem using monitored data from an actual cable-stayed bridge

Abstract: A structural health monitoring (SHM) system provides an efficient way to diagnose the condition of critical and large-scale structures such as long-span bridges. With the development of SHM techniques, numerous condition assessment and damage diagnosis methods have been developed to monitor the evolution of deterioration and long-term structural performance of such structures, as well as to conduct rapid damage and post-disaster assessments. However, the condition assessment and the damage detection methods described in the literature are usually validated by numerical simulation and/or laboratory testing of small-scale structures with assumed deterioration models and artificial damage, which makes the comparison of different methods invalid and unconvincing to a certain extent. This paper presents a full-scale bridge benchmark problem organized by the Center of Structural Monitoring and Control at the Harbin Institute of Technology. The benchmark bridge structure, the SHM system, the finite element model of the bridge, and the monitored data are presented in detail. Focusing on two critical and vulnerable components of cable-stayed bridges, two benchmark problems are proposed on the basis of the field monitoring data from the full-scale bridge, that is, condition assessment of stay cables (Benchmark Problem 1) and damage detection of bridge girders (Benchmark Problem 2). For Benchmark Problem 1, the monitored cable stresses and the fatigue properties of the deteriorated steel wires and cables are presented. The fatigue life prediction model and the residual fatigue life assessment of the cables are the foci of this problem. For Benchmark Problem 2, several damage patterns were observed for the cable-stayed bridge. The acceleration time histories, together with the environmental conditions during the damage development process of the bridge, are provided. Researchers are encouraged to detect and to localize the damage and the damage development process. All the datasets and detailed descriptions, including the cable stresses, the acceleration datasets, and the finite element model, are available on the Structural Monitoring and Control website (http://smc.hit.edu.cn). Copyright © 2013 John Wiley & Sons, Ltd.