Section: New Results

Computer virology: behavioral analysis

Participants : Isabelle Gnaedig, Jean-Yves Marion, Philippe Beaucamps.

Our study on behavioural malware detection has been continued. We have been developing an approach detecting suspicious schemes on an abstract representation of the behavior of a program, by abstracting program traces, rewriting given subtraces into abstract symbols representing their functionality. Considering abstract behaviors allows us to be implementation-independent and robust to variants and mutations of malware. Suspicious behaviors are then detected by comparing trace abstractions to reference malicious behaviors.

Last year, we had proposed to abstract trace automata by rewriting them with respect to a set of predefined behavior patterns defined as a regular language described by a string rewriting system  [35] . We have increased the power of our approach on two aspects. We fist have modified the abstraction mechanism, keeping the abstracted patterns in the rewritten traces, by just marking them. This now allows us to handle interleaved patterns. Second, we have extended the rewriting framework to express data constraints on action parameters by using term rewriting systems. An important consequence is that, unlike in  [35] , using the data-flow, we can now detect information leaks in order to prevent unauthorized disclosure or modifications of information [28] .

The previous approach has also been extended to a probabilistic model of rewriting, in order to express uncertainty in the behavior pattern recognition. All these results on detection of malware by behavior abstraction have been given in the PhD thesis of Philippe Beaucamps, directed by Isabelle Gnaedig and Jean-Yves Marion, and defended 14 November, 2011 [11] .