Showing papers on "Benchmark (computing) published in 1994"

Automated learning of decision rules for text categorization

Abstract: We describe the results of extensive experiments using optimized rule-based induction methods on large document collections. The goal of these methods is to discover automatically classification patterns that can be used for general document categorization or personalized filtering of free text. Previous reports indicate that human-engineered rule-based systems, requiring many man-years of developmental efforts, have been successfully built to “read” documents and assign topics to them. We show that machine-generated decision rules appear comparable to human performance, while using the identical rule-based representation. In comparison with other machine-learning techniques, results on a key benchmark from the Reuters collection show a large gain in performance, from a previously reported 67% recall/precision breakeven point to 80.5%. In the context of a very high-dimensional feature space, several methodological alternatives are examined, including universal versus local dictionaries, and binary versus frequency-related features.

PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms

Abstract: Proben1 is a collection of problems for neural network learning in the realm of pattern classification and function approximation plus a set of rules and conventions for carrying out benchmark tests with these or similar problems. Proben1 contains 15 data sets from 12 different domains. All datasets represent realistic problems which could be called diagnosis tasks and all but one consist of real world data. The datasets are all presented in the same simple format, using an attribute representation that can directly be used for neural network training. Along with the datasets, Proben1 defines a set of rules for how to conduct and how to document neural network benchmarking. The purpose of the problem and rule collection is to give researchers easy access to data for the evaluation of their algorithms and networks and to make direct comparison of the published results feasible. This report describes the datasets and the benchmarking rules. It also gives some basic performance measures indicating the difficulty of the various problems. These measures can be used as baselines for comparison.

Compiler optimizations for improving data locality

Abstract: In the past decade, processor speed has become significantly faster than memory speed. Small, fast cache memories are designed to overcome this discrepancy, but they are only effective when programs exhibit data locality. In this paper, we present compiler optimizations to improve data locality based on a simple yet accurate cost model. The model computes both temporal and spatial reuse of cache lines to find desirable loop organizations. The cost model drives the application of compound transformations consisting of loop permutation, loop fusion, loop distribution, and loop reversal. We demonstrate that these program transformations are useful for optimizing many programs.To validate our optimization strategy, we implemented our algorithms and ran experiments on a large collection of scientific programs and kernels. Experiments with kernels illustrate that our model and algorithm can select and achieve the best performance. For over thirty complete applications, we executed the original and transformed versions and simulated cache hit rates. We collected statistics about the inherent characteristics of these programs and our ability to improve their data locality. To our knowledge, these studies are the first of such breadth and depth. We found performance improvements were difficult to achieve because benchmark programs typically have high hit rates even for small data caches; however, our optimizations significantly improved several programs.

Detecting Abnormal Operating Performance: The Empirical Power and Specification of Test Statistics

Abstract: Many recent studies have analyzed the impact that corporate events or managerial decisions have on operating performance. In these studies, researchers face many methodological choices. This paper analyzes the effect that three dimensions of choice have on the specification and power of test-statistics designed to detect abnormal operating performance: (1) the selection of a performance measures (e.g. return on assets or return on sales), (2) the selection of a statistical test (e.g., parametric t-statistic or non-parametric Wilcoxon T*), and (3) the selection of a performance benchmark (where we evaluate nine different performance benchmarks). On the first two dimensions (choice of performance measures and statistical test) we generally find little difference in the specification and power of test-statistics. However, on the third dimension (choice of performance benchmark), only one benchmark -- matching sample firms to firms with the same two-digit SIC code and similar past performance -- is well-specified. In the conclusion, we provide specific recommendations about the choice of performance measure, test-statistic, and benchmark.

On area/depth trade-off in LUT-based FPGA technology mapping

Abstract: In this paper, we study the area and depth trade-off in lookup-table (LUT) based FPGA technology mapping. Starting from a depth-optimal mapping solution, we perform a sequence of depth relaxation operations and area-minimizing mapping procedures to produce a set of mapping solutions for a given design with smooth area and depth trade-off. As the core of the area minimization step, we have developed a polynomial time optimal algorithm for computing an area-minimum mapping solution without node duplication for a K-bounded general Boolean network, which makes a significant step towards complete understanding of the general area minimization problem in FPGA technology mapping. The experimental results on MCNC benchmark circuits show that our solution sets outperform the solutions produced by most existing mapping algorithms in terms of both area and depth minimization. >

Switching Activity Analysis Considering Spatioternporal Correlations

Abstract: This work presents techniques for computing the switching activities of all circuit nodes under pseudorandom or biased input sequences and assuming a zero delay mode of operation. Complex spatiotemporal correlations among the circuit inputs and internal nodes are considered by using a lag-one Markov Chain model. Evaluations of the model and a comparative analysis presented for benchmark circuits demonstrates the accuracy and the practicality of the method. The results presented in this paper are useful in power estimation and low power design.

An Exact Algorithm for Selecting Partial Scan Flip-Flops

Abstract: We develop an exact algorithm for selecting flip-flops in partial scan designs to break all feedback cycles. The main ideas that allowus to solve this hard problemexactly for large, practical instances are - graph transformations, a partitioning scheme used in the branch and bound procedure, and pruning techniques based on an integer linear programming formulation of the minimum feedback vertex set (MFVS) problem.We have obtained optimum solutions for the ISCAS '89 benchmark circuits and several production VLSI circuits within reasonable computation time. For example, the optimal number of scan flip-flops required to eliminate all cycles except self-loops in the circuit s38417 is 374. This optimal solution was obtained in 32 CPU seconds on a SUN Sparc 2 workstation.

Link-time optimization of address calculation on a 64-bit architecture

Abstract: Compilers for new machines with 64-bit addresses must generate code that works when the memory used by the program is large. Procedures and global variables are accessed indirectly via global address tables, and calling conventions include code to establish the addressability of the appropriate tables. In the common case of a program that does not require a lot of memory, all of this can be simplified considerably, with a corresponding reduction in program size and execution time.We have used our link-time code modification system OM to perform program transformations related to global address use on the Alpha AXP. Though simple, many of these arewhole-program optimizations that can be done only when we can see the entire program at once, so link-time is an ideal occasion to perform them.This paper describes the optimizations performed and shows their effects on program size and performance. Relatively modest transformations, possible without moving code, improve the performance of SPEC benchmarks by an average of 1.5%. More ambitious transformations, requiring an understanding of program structure that is thorough but not difficult at link-time, can do even better, reducing program size by 10% or more, and improving performance by an average of 3.8%.Even a program compiled monolithically with interprocedural optimization can benefit nearly as much from this technique, if it contains statically-linked pre-compiled library code. When the benchmark sources were compiled in this way, we were still able to improve their performance by 1.35% with the modest transformations and 3.4% with the ambitious transformations.

Results of the holonic control system benchmark at KU Leuven

Abstract: This paper surveys the results of a benchmark performed in the framework of IMS test case No. 5, entitled: 'Holonic Manufacturing Systems: System Components of Autonomous Modules and their Distributed Control.' This test case is a one-year project that puts forward a new paradigm for manufacturing-the holonic manufacturing system. The benchmark compares the performance of a prototype holonic control system with both a hierarchical and a heterarchical counterpart. >

Automated benchmarking with self customization

Abstract: The method enables realtime establishment and maintenance of a standard of operation for a data communications network. The process begins by monitoring the network over some period of time to build benchmark data sets. The benchmark data sets contain a standard of operation for the network, which is historically categorized by either traffic type or activity. This standard of operation is accumulated by the intelligent monitoring facilities. After a period of accumulation, the benchmark is used to determine whether data taken from current monitoring activity indicates normal network behavior. Network monitoring information is analyzed using criteria in modules that have an interface to an expert system. The criteria modules evaluate the current monitored data against the prior benchmark collected data. The criteria module determines if the current network operating characteristics are outside the bounds of normal behavior. If they are, then alerts and logs of information can be sent to the expert system. The expert system can perform network control routing changes, or close down applications, or allocate additional bandwidth as required. In addition, the expert system can modify the characterization of the currently monitored data for historical purposes by supplying information to the benchmark manager regarding traffic types, activity, heuristic accuracy and changes between current and past behavior. In this manner, auto benchmarking can be accomplished with self customization, in an improved manner.

An automatic test pattern generator for large sequential circuits based on Genetic Algorithms

Abstract: This paper is concerned with the question of automated test pattern generation for large synchronous sequential circuits and describes an approach based on Genetic Algorithms suitable for even the largest benchmark circuits, together with a prototype system named GATTO. Its effectiveness (in terms of result quality and CPU time requirements) for circuits previously unmanageable is illustrated. The flexibility of the new approach enables users to easily trade off fault coverage and CPU time to suit their needs.

An engineering database benchmark

Abstract: Performance is a major issue in the acceptance of object-oriented and extended relational database systems aimed at engineering applications such as Computer-Aided Software Engineering (CASE) and Computer-Aided Design (CAD). Because traditional database system benchmarks (Bitton, DeWitt, & Turbyfill [BITT84], Anon et. al. [ANON85], TPC [TPC89]) do not measure the performance of features essential to engineering applications, we designed an engineering database benchmark. The benchmark, named EDB (for Engineering Database Benchmark), has been run on a dozen database products and prototypes. This chapter describes EDB and the results obtained running it on a relational and on an object-oriented DBMS. This chapter derives from a paper by Cattell and Skeen [CATT91]. EDB is a redesign of an earlier, more primitive benchmark described in Rubenstein, Kubicar, and Cattell [RUBE1987].

A status report on the OO7 OODBMS benchmarking effort

Abstract: The OO7 Benchmark was first published in 1993, and has since found a home in the marketing literature of various object-oriented database management system (OODBMS) vendors. The OO7 Benchmark (as published) was the initial result of an ongoing OODBMS performance evaluation effort at the University of Wisconsin. This paper provides an update on the status of the effort on two fronts: single-user and multi-user. On the single-user front, we review and critique the design of the initial OO7 Benchmark. We discuss some of its faults, the reasons for those faults, and things that might be done to correct them. On the multi-user front, we describe our current work on the development of a multi-user benchmark for OODBMSs. This effort includes changes and extensions to the OO7 database and the design of a family of interesting multi-user workloads.

QuickStore: a high performance mapped object store

Abstract: This paper presents, QuickStore, a memory-mapped storage system for persistent C++ built on top of the EXODUS Storage Manager. QuickStore provides fast access to in-memory objects by allowing application programs to access objects via normal virtual memory pointers. The paper also presents the results of a detailed performance study using the OO7 benchmark. The study compares the performance of QuickStore with the latest implementation of the E programming language. These systems exemplify the two basic approaches (hardware and software) that have been used to implement persistence in object-oriented database systems. Both systems use the same underlying storage manager and compiler allowing us to make a truly apples-to-apples comparison of the hardware and software techniques.

Performance estimation of multistreamed, superscalar processors

Abstract: Multistreamed processors can significantly improve processor throughput by allowing interleaved execution of instructions from multiple instruction streams. We present an analytical modeling technique to evaluate the effect of dynamically interleaving additional instruction streams within superscalar architectures. Using this technique, estimates of the instructions executed per cycle (IPC) for a processor architecture are quickly calculated given simple descriptions of the workload and hardware characteristics. To validate this technique, estimates of the SPEC89 benchmark suite obtained from the model are compared to results from a hardware simulator. Our results show that the technique produces accurate estimates with an average deviation of /spl sim/4% from the simulation results. Finally, we demonstrate that as the number of functional units increases, multistreaming is an effective technique to exploit these additional resources. >

Application of simple genetic algorithms to sequential circuit test generation

Abstract: In this work we investigate the effectiveness of genetic algorithms (GAs) in the test generation process. We use simple GAs to generate populations of candidate test vectors and to select the best vector to apply in each time frame. A sequential circuit fault simulator is used to evaluate the fitness of each candidate vector, allowing the test generator to be used for both combinational and sequential circuits. We experimented with various GA parameters, namely population size, number of generations, mutation rate, and selection and crossover schemes. For the ISCAS85 combinational benchmark circuits, 100% of testable faults were detected in six of the ten circuits used, and very compact test sets were generated. Good results were obtained for many of the ISCAS89 sequential benchmark circuits, and execution times were significantly lower than in a deterministic test generator in most cases. >

PRISC: programmable reduced instruction set computers

Abstract: This thesis introduces Programmable Reduced Instruction Set Computers (PRISC) as a new class of general-purpose computers PRISC use RISC techniques as a base, but in addition to the conventional RISC instruction resources, PRISC offer hardware programmable resources which can be configured based on the needs of a particular application This thesis presents the architecture, operating system, and programming language compilation techniques which are needed to successfully build PRISC Performance results are provided for the simplest form of PRISC--a RISC microprocessor with a set of programmable functional units consisting of only combinational functions Results for the SPECint92 benchmark suite indicate that an augmented compiler can provide a performance improvement of 22% over the underlying RISC computer with a hardware area investment less than that needed for a 2 kilobyte SRAM In addition, active manipulation of the source code leads to significantly higher local performance gains (250%-500%) for general abstract data types such as short-set vectors, hash tables, and finite state machines Results on end-user applications that utilize these data types indicate a performance gain from 32%-213%

A comparison of message passing and shared memory architectures for data parallel programs

Abstract: Shared memory and message passing are two opposing communication models for parallel multicomputer architectures. Comparing such architectures has been difficult, because applications must be hand-crafted for each architecture, often resulting in radically different sources for comparison. While it is clear that shared memory machines are currently easier to program, in the future, programs will be written in high-level languages and compiled to the specific parallel target, thus eliminating this difference.In this paper, we evaluate several parallel architecture alternatives --- message passing, NUMA, and cachecoherent shared memory --- for a collection of scientific benchmarks written in C*, a data-parallel language. Using a single suite of C* source programs, we compile each benchmark and simulate the interconnect for the alternative models. Our objective is to examine underlying, technology-independent costs inherent in each alternative. Our results show the relative work required to execute these data parallel programs on the different architectures, and point out where some models have inherent advantages for particular data-parallel program styles.

Parallel Spectral Transform Shallow Water Model: a runtime-tunable parallel benchmark code

Abstract: Fairness is an important issue when benchmarking parallel computers using application codes. The best parallel algorithm on one platform may not be the best on another. While it is not feasible to re-evaluate parallel algorithms and reimplement large codes whenever new machines become available, it is possible to embed algorithmic options into codes that allow them to be "tuned" for a particular machine without requiring code modifications. We describe a code in which such an approach was taken. PSTSWM was developed for evaluating parallel algorithms for the spectral transform method in atmospheric circulation models. Many levels of runtime-selectable algorithmic options are supported. We discuss these options and our evaluation methodology. We also provide empirical results from a number of parallel machines, indicating the importance of tuning for each platform before making a comparison. >

A new dynamic test vector compaction for automatic test pattern generation

Abstract: A new approach for dynamic test vector compaction, for combinational logic circuits, called COMPACT, is proposed. A new data structure of test vectors permits easy verification of compactability between test vectors with minimal memory requirements. Experimental results obtained by adding the proposed algorithm to a simple PODEM program and applying it to the ISCAS-85 benchmark circuits are presented. The resulting test vector reduction is up to 40% for small circuits and around 50% for the large circuits (over 1000 gates). >

A new approach to I/O performance evaluation: self-scaling I/O benchmarks, predicted I/O performance

Abstract: Current I/O benchmarks suffer from several chronic problems: they quickly become obsolete; they do not stress the I/O system; and they do not help much in understanding I/O system performance. We propose a new approach to I/O performance analysis. First, we propose a self-scaling benchmark that dynamically adjusts aspects of its workload according to the performance characteristic of the system being measured. By doing so, the benchmark automatically scales across current and future systems. The evaluation aids in understanding system performance by reporting how performance varies according to each of five workload parameters. Second, we propose predicted performance, a technique for using the results from the self-scaling evaluation to estimate quickly the performance for workloads that have not been measured. We show that this technique yields reasonably accurate performance estimates and argue that this method gives a far more accurate comparative performance evaluation than traditional single-point benchmarks. We apply our new evaluation technique by measuring a SPARCstation 1+ with one SCSI disk, an HP 730 with one SCSI-II disk, a DECstation 5000/200 running the Sprite LFS operating system with a three-disk disk array, a Convex C240 minisupercomputer with a four-disk disk array, and a Solbourne 5E/905 fileserver with a two-disk disk array.

Interacting multiple model algorithm for solution to benchmark problem for tracking maneuvering targets

Abstract: This paper presents a solution to a benchmark problem for tracking maneuvering targets. The benchmark problem involves beam pointing control of a phased array (i.e., an agile beam) radar against highly maneuvering targets. The proposed solution utilizes an interacting multiple model (IMM) algorithm that includes a constant velocity model, a constant thrust model, and a constant speed turn model. The output error covariance of the IMM algorithm is used to compute the time for the next measurement so that a given level of tracking performance is maintained. Using this on-line measure of tracking performance automatically takes into account target range, target maneuvers, missed detections, and strength of the returns. A testbed simulation program that includes the effects of target amplitude fluctuations, beamshape, missed detections, finite resolution, target maneuvers, and track loss is used to evaluate the performance of the proposed algorithm. The `best' tracking algorithm as defined by the benchmark problem is the one that requires a minimum number or radar dwells while satisfying a constraint of 4% on the maximum number of lost tracks. The proposed technique lost less than 2% of the tracks and provided average sample periods of 3.6 s for the commercial aircraft trajectory and 1.9 s for targets maneuvering with as much as 7 g's.

An ultra-high precision benchmark for validation of planar electromagnetic analyses

Abstract: A stripline standard is applied to the validation of planar electromagnetic analysis. Since an exact theoretical expression is available for stripline, a benchmark can be specified to the accuracy to which the expressions can be evaluated. Data for the benchmark accurate to about 10/sup -8/ is provided. A definition for an error metric appropriate for use with the benchmark is illustrated. A means of calculating a precise value of analysis error using the error metric is described. A first order numerical value for the residual analysis error can also be obtained from the calculated S-parameters by inspection. The benchmark can be applied to any planar electromagnetic analysis capable of analyzing stripline. Example results, illustrating absolute convergence of an analysis to 0.05%, are provided. >

Synthesis of sigma-pi neural networks by the breeder genetic programming

Abstract: Genetic programming has been successfully applied to evolve computer programs for solving a variety of interesting problems. The breeder genetic programming (BGP) method has Occam's razor in its fitness measure to evolve minimal size multilayer perceptrons. In this paper, we apply the method to synthesis of sigma-pi neural networks. Unlike perceptron architectures, sigma-pi networks use product units as well as summation units to build higher-order terms. The effectiveness of the method is demonstrated on benchmark problems. Simulation results on noisy data suggest that BGP not only improves the generalization performance, but it can also accelerate the convergence speed. >

A solution to an automotive control system benchmark

Abstract: The Society of Automotive Engineers (SAE) has recently published a set of requirements and a control benchmark which is able to compare the effectiveness of new protocol proposals for safety-critical automotive systems. This paper presents a solution to this benchmark problem that is based on the Time Triggered Protocol (TTP). TTP integrates all services required for the implementation of fault-tolerant hard real-time systems, while trying to minimize the bandwidth requirements. >

On the initialization of sequential circuits

Abstract: A general method for determining whether a certain design is initializable, and for generating its initialization sequence, is presented. This method is based on structural decomposition of the circuit, and can handle both logical (using X-value simulation) and functional initializabilities. Results for some benchmark circuits are also presented.