With the rapid development of information storage and networking technologies, quintillion bytes of data are generated every day from social networks, business transactions, sensors, and many other domains. The increasing data volumes impose significant challenges to traditional data analysis tools in storing, processing, and analyzing these extremely large-scale data. For decades, hashing has been one of the most effective tools commonly used to compress data for fast access and analysis, as well as information integrity verification. Hashing techniques have also evolved from simple randomization approaches to advanced adaptive methods considering locality, structure, label information, and data security, for effective hashing. This survey reviews and categorizes existing hashing techniques as a taxonomy, in order to provide a comprehensive view of mainstream hashing techniques for different types of data and applications. The taxonomy also studies the uniqueness of each method and therefore can serve as technique references in understanding the niche of different hashing mechanisms for future development.

Hashing Techniques: A Survey and Taxonomy

https://digitalcommons.newhaven.edu/cgi/viewcontent.cgi?article=1036&context=electricalcomputerengineering-facpubs

A Cyber Forensics Needs Analysis Survey: Revisiting the Domain's Needs a Decade Later

https://digitalcommons.newhaven.edu/cgi/viewcontent.cgi?article=1061&context=electricalcomputerengineering-facpubs

Automated evaluation of approximate matching algorithms on real data

Bootstrapping: For the bootstrapping experiment, we use an LSTM [16] with 256 hidden units as meta network. The inputs to the meta network include reward Rt+1, discount γt+1, value from the next state v(St+1), which were fed in reverse-time order into the LSTM. The agent network is a two layer MLP with 256 hidden units. Both agent optimiser and the meta optimiser are RMSProp [35], where learning rate is 1e−3 and meta learning rate is 1e−4. M = 5 inner updates are performed to the agent then the metagradient is obtained. We perform 10 independent runs with 10 different random seeds and report average performance with standard deviation.

Implementation details

Fuzzy hashing algorithms provide a convenient way of summarizing in a compact form the content of files, and of looking for similarities between them. Because of this, they are widely used in the security and forensics communities to look for similarities between binary program files; one version of them, ssdeep, is the de facto standard to share information about known malware. Fuzzy hashes are quite pervasive, but no study so far answers conclusively the question of which (if any) fuzzy hashing algorithms are suited to detect similarities between programs, where we consider as similar those programs that have code or libraries in common. We measure how four popular algorithms perform in different scenarios: when they are used to correlate statically-compiled files with the libraries they use, when compiled with different flags or different compilers, and when applied to programs that share a large part of their source code. Perhaps more importantly, we provide interpretations that explain the reasons why results vary, sometimes widely, among apparently very similar use cases. We find that the low-level details of the compilation process, together with the technicalities of the hashing algorithms, can explain surprising results such as similarities dropping to zero with the change of a single assembly instruction. More in general, we see that ssdeep, the de facto standard for this type of analysis, performs definitely worse than alternative algorithms; we also find that the best choice of algorithm to use varies depending on the particularities of the use case scenario.

Beyond Precision and Recall: Understanding Uses (and Misuses) of Similarity Hashes in Binary Analysis

This document provides a definition of and terminology for approximate matching Approximate matching is a promising technology designed to identify similarities between two digital artifacts It is used to find objects that resemble each other or to find objects that are contained in another object This can be very useful for filtering data for security monitoring, digital forensics, or other applications The purpose of this document is to provide a definition and terminology to describe approximate matching in order to promote discussion, research, tool development and tool acquisition

/pdf/approximate-matching-definition-and-terminology-55c8bfp8uz.pdf

Michael McCarrin

Papers

Approximate Matching: Definition and Terminology