HAL Id: hal-00762252
https://hal.inria.fr/hal-00762252
Submitted on 10 Dec 2012
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Abstraction-based Malware Analysis Using Rewriting
and Model Checking
Philippe Beaucamps, Isabelle Gnaedig, Jean-Yves Marion
To cite this version:
Philippe Beaucamps, Isabelle Gnaedig, Jean-Yves Marion. Abstraction-based Malware Analysis Using
Rewriting and Model Checking. ESORICS - 17th European Symposium on Research in Computer
Security - 2012, Sep 2012, Pisa, Italy. pp.806-823, �10.1007/978-3-642-33167-1�. �hal-00762252�
Abstraction-based Malware Analysis Using
Rewriting and Model Checking
Philippe Beaucamps
1
, Isabelle Gnaedig
2
, Jean-Yves Marion
1
1
Universit´e de Lorraine, LORIA, UMR 7503, Vandoeuvre-l`es-Nancy, F-54506, France
2
Inria, Villers-l`es-Nancy, F-54600, France
{Philippe.Beaucamps,Isabelle.Gnaedig,Jean-Yves.Marion}@loria.fr
Abstract. We propose a formal approach for the detection of high-level
malware behaviors. Our technique uses a rewriting-based abstraction
mechanism, producing abstracted forms of program traces, independent
of the program implementation. It then allows us to handle similar be-
haviors in a generic way and thus to be robust with respect to variants.
These behaviors, defined as combinations of patterns given in a signa-
ture, are detected by model-checking on the high-level representation of
the program. We work o n unbounded sets of traces, which makes our
technique useful not only for dynamic analysis, considering one trace at
a time, but also for static analysis, considering a set of traces inferred
from a control flow graph. Abstrac ting traces with rewriting systems on
first order terms with variables allows us in particular to model dataflow
and to detect information leak.
Keywords: Malware, behavioral detection, behavior abstraction, trace,
term rewriting, model checking, first order temporal logic, finite state
automaton, formal language.
1 Introduction
Behavior analysis was introduced by Cohen’s seminal work [11] to detect mal-
ware and in particular unknown malware. In general, a behavior is described by
a sequence of system calls and recognition uses the formalism of finite state au-
tomata [22, 26, 24, 6]. New approaches have been proposed recently. In [18, 27],
malicious behaviors ar e specified by temporal logic formulas with parameters
and detection is carried out by model-checking. However, th ese approaches are
tightly dependent on the way malicious actions are realized: using any oth er
system facility to realize an action allows a malware to go undetected. This has
motivated yet another approach where a malicious behavior is specified as a
combination of high-level actions, in order to be independent from the way these
actions are realized and to only consider their effect on a system. In [23] and
in [3], a captured execution trace is transformed into a higher-level represen-
tation capturing its semantic m eaning, i.e., the trace is first abstracted before
being compared to a malicious behavior. In [17], the authors propose to use
attribute automata, at the price of an exponential time complexity detection.
These dynamic abstraction-based approaches, though they can detect unknown
viruses whose execution traces exhibit known malicious behaviors, only deal with
a single execution trace.
In this paper, we propose a formal approach for high-level behavior analysis,
with the following features. Underpinned by language theory, term rewriting and
first-order temporal logic, it allows us to determine whether a program exhibits
a high-level behavior. Detection is achieved in two steps. First, traces of the pro-
gram are abstracted in order to reveal the sequences of high-level functionalities
they realize. Then, abstracted traces are compared with the behavior formula,
using usual model-checking techniques. Functionalities have parameters repre-
senting the manipulated data, so our formalism is adapted to the protection
against generic threats like the leak of sensitive information.
Our goal here is not to provide a ready-made software to detect behaviors, but
to propose a formal framewok emphasizing fundamental detection mechanisms,
which are independent of implementation-based solutions.
Our approach has two main characteristics. First, we work on an unbounded
set of traces representing the behavior of a program, in order to consider a more
complete representation of the program than with a single trace. To deal with
the infinity of the set of traces, we restrict to regular sets an d safely approximate
the set of abstract traces, so that we detect in linear time whether a program
exhibits a given behavior. Second, we work on abstract forms of traces, in or-
der to only keep the essence of the functions performed by the program, to be
independent of their possible implementations and to be generic with respect
to behavior mutations. Behavior components are abstracted in program traces,
by identifying known functionalities and marking them by inserting abstract
functionality symbols.
By working on sets of tr aces, which may consist of a single trace as well
as of an unbounded numb er of traces, our approach may be used not only for
classical, dynamic behavior analysis, but also for static behavior analysis i.e.,
behavior analysis in a static analysis setting.
Static behavior analysis by abstraction is more challenging than its dynamic
counterpart because, precisely, this approach needs to abstract a program behav-
ior potentially representing an infinite set of execution traces. The construction
of an exhaustive representation of a program behavior is an intractable prob-
lem in general: in particular, a program flow may not be easily followed due to
indirect jumps, and a program may use complex code protection, for instance
by dynamically modifying its code or by using obfuscation. Self modification is
usually tackled by emulating the program long enough to deactivate most code
protections. Indirect jumps and obfuscation are usually handled by abstract in-
terpretation [25, 19] or symbolic execution [7].
Static behavior analysis has many advantages and applications. First, it al-
lows us to analyze the behavior of a program in a more exhaustive way, as it
analyzes the unbounded set of the program execution traces, or an approxima-
tion of it. Second, static behavior analysis can complement classical, dynamic,
behavior analysis with an analysis of the future behavior, to prevent damages
when some critical point is reached in an execution.
An interesting application of static behavior analysis is the audit of pro-
grams in high-level technologies, like mobile applications, browser extensions,
web page scripts, .NET or Java programs. Auditing these programs is complex
and mostly manual, resulting in highly publicized infections [2, 1]. In this con-
text, static analysis can provide an appropriate help, because it is usually easier
than for usual programs, especially when additionally enforcing a security pol-
icy (e.g. p rohibiting self-modification [28]) or when enforcing strict development
guidelines (e.g. for iPhone applications).
To our knowledge, the use of behavior abstraction on top of static behavior
analysis has not been investigated so far. As our detection mechanism relies on
satisfaction of temporal logic formulas, it is akin to model checking [21], for which
there already exist numerous frameworks and tools [16, 14, 8]. Th e specificity of
our approach, however, is that, rather than being applied on the set of program
traces, verification is applied on the set of abstract forms of these traces, which is
not computable in general. Accordingly, we identify a property of practical high-
level behaviors allowing us to approximate this set, in a sound and complete way
with respect to detection, and then to apply classical verification techniques.
Our abstraction framework can be used in two scenarios:
– Detection of given behaviors: signatures of given high-level behaviors are ex-
pressed in terms of abstract functionalities. Given some program, we then
assess whether one of its execution traces exhibits a s equence of known func-
tionalities, in a way specific to one of the given behaviors. This can be applied
to detection of suspicious behaviors. Although detection of such suspicious
behaviors may not suffice to label a program as malicious, it can be used to
supplement existing detection techniques with additional decision criteria.
– Analysis of programs: abstraction provides a simple and high-level represen-
tation of a program behavior, which is more suitable than the original traces
for manual analysis, or for analysis of behavior similarity with known be-
haviors, etc. For instance, it could be used to detect not necessarily harmful
behaviors, in order to get a basic understanding of the program and to fur-
ther investigate if deemed necessary. It could also be used to automatically
discover sequences of high-level functionalities and their dataflow dependen-
cies, exhibited by a program.
Previous work. In [4], we already proposed to abstract program sets of traces
with respect to behavior patterns, for detection and analysis. We tested our
approach on samples of malicious programs collected using a honeypot
3
and
identified using Kaspersky Antivirus. These samples belonged to known malware
families, like Allaple, Virut, Agent, R bot, Afcore and Mimail. Most of them were
successfully matched to our malware database.
3
The honeypot of the Loria’s High Security Lab: http://lhs.loria.fr
But patterns were defined by string rewriting systems, which did not allow
the actions composing a trace to have parameters, precluding dataflow analysis.
Moreover, abstraction rules replaced identified patterns by abstraction symbols
in the original trace, precluding a further detection of patterns interleaved with
the rewritten ones.
The formalism proposed in this paper addresses both issues: first, we handle
interleaved patterns by keeping the identified patterns when abstracting them.
Second, we extend the rewriting framework to express data constraints on action
parameters by using term rewriting systems. An important consequence is that,
unlike in [4], using the dataflow, we can detect information leaks in order to
prevent unauthorized disclosure or modifications of information.
2 Background
Term Algebras. Let S = {T race, Action, Data} be a set of sorts, F = F
t
∪ F
a
∪F
d
be a finite S-sorted signature, where F
t
, F
a
, F
d
are mutually distinct and:
– F
t
= {ǫ, ·} is the set of the trace constructors, where ǫ :→ T race denotes
the empty trace, . has profile Data T race → T race;
– F
a
is a set of function symbols or constants, with profile Data
n
→ Action,
n ∈ N, describing actions;
– F
d
is a set of data constructors, with profile → Data or Data
n
→ Data,
n ∈ N.
Let N
∗
+
be the set of finite strings of positive natural numbers, called positions.
The empty string is denoted by λ, and u ≤ v means that u is prefix of v. Let X
be a set of S-sorted variables. A S-sorted term over (F, X) is a partial function
t : N
∗
+
→ F ∪ X, such that the domain of definition of t, denoted by Pos(t),
is finite and satisfies, for w ∈ N
∗
+
and i ∈ N: (1) wi ∈ Pos(t) ⇒ w ∈ Pos(t),
(2) w ∈ Pos(t) ⇒ t(w) ∈ F ∪ X. Pos(t) is called the set of positions of t. We
denote by T (F, X) (resp. T (F)) the set of S-sorted terms over ( F, X) (resp. the
set of finite ground terms over F). For any sort s ∈ S, and any of the above sets
of terms T we denote by T
s
the restriction of T to terms of sort s and by X
s
the subset of variables of X of sort s. For a term t with p ∈ Pos(t), we denote
by t|
p
the subterm of t at position p. We denote by t[t
′
]
p
the term obtained by
replacing by t
′
the subterm at position p in t. We use the abbreviated notation
x for variables x
1
, . . . , x
n
. So x ∈ X stands for x
1
, . . . , x
n
∈ X, and if f ∈ F is
a symbol of arity n ∈ N, we denote by f (
x) the term f (x
1
, . . . , x
n
).
The elements of T
Trace
(F) are called traces, the elements of T
Action
(F) are
called actions. We distinguish the sort Action f rom the sort Trace but, for a
sake of readability, we may denote by a the trace · (a, ǫ), for some action a.
Similarly, we use the · symbol with infix notation and right associativity, and
ǫ is understood when the context is unambiguous. For instance, if a, b, c are
actions, a · b · c denotes the trace · (a, · ( b , · (c, ǫ)) ).
We partition F
a
in a set Σ of symbols, denoting concrete program-le vel ac-
tions, and a set Γ , denoting abstract actions identifying abstracted functional-
ities. To construct purely concrete (resp. abstract) terms, we use F
Σ
= F \ Γ