What are the popular machine learning unsupervised approaches to learning binary data representations?

Question

This is somewhat open-ended question, so bear with me please.

Be it a binary payload that you pull from network packet, firmware blob you pull of some EPROM or data intercepted straight from physical bus - does anybody here use machine learning models to learn data representation?

The problem here is that we have a large sequence(s) of binary data of known length which has some kind of a structure (think stream of IP packets) which is yet to be understood.

Assuming we have enough of this data, we can deduce data representations in unsupervised manner, e.g. if we have pcap capture of a network packet flow on ethernet network, we can deduce what ethernet header is, that it is typically followed by something like IP header, then UDP/TCP header and some paylond in the end.

You could replace "classifier" with whatever else we use those ML models (in this case probably RNN-GRU/LSTM) for nowadays (apart from classification it could generate fake traffic etc). But the point is that unlike with common ML domains such as natural language processing, I'm not aware of any model that can be used to replace manual parsing. Most popular NLP model to learn what words mean in context of a sequence nowadays is word2vec, but is there anything like that for representing binary sequences?

PS: I used example in the image just to demonstrate the problem, that could very well be binary code of unknown architecture, USB request block or anything else, point being that it is typically highly structured, opaque and sequential binary input.

I guess the real question is, does anybody use ML for reverse engineering? If so, do you mind sharing your experiences?

Answer 1

Inferring the structure of an unknown binary stream is a really difficult task, that I'm aware of only a few attempts at tackling:

1. Pulsar is an example project but I suspect it is not really mature.
2. Netzob is another project, and it seems more complete.
3. Radamsa is a third project. looks more complete than Pulsar and more actively maintained than both.

I do not believe existing "machine learning" models and approaches are currently capable of statistically inferring structural information about binary data. You'll have to resort to more basic statistical inference, and that process should probably be designed from beginning to end with the end goal in mind (seems to be "malicious classification").

Most approaches nowadays do not tackle the task of protocol inference in the process of malware classification (one could also make a few arguments against any advantage of doing so).

In comparison to malware classification, I would say the task of protocol structure inference is more difficult to achieve and I would advise reevaluating if this is indeed a requirement you want to attempt meeting.

Full disclosure: I'm employed by a "next gen AV" that makes use of ML for malware classification and have been tinkering with protocol inference in the past.

Answer 2

First, the more samples of discrete messages you have the better.

Second, a one algorithm fits all approach isn't going to work. Use an ensemble of small simple recognizers.

Third, it's coming from memory, so start by learning the memory representations for a specific thing, for example unint32.

Fourth, the most difficult part of this task is having pure data. You can be comparing string bytes to int bytes and get nowhere. If you have pure string bytes or pure float bytes, you'll make progress.