Can AI be used to write better decompilers/disassemblers? [closed]

Question

In the past for years I used some disassembler and tried some decompiler, nowadays there's so much talking and stuff about deep learning and AI, I wonder if some can be used with those tasks (given some human training) and if there's some tool using it already.

Answer 1

First, you cannot answer to this question without having to define what you mean by AI... Because this is probably the worst name for a domain in computer science. Where people think about "programs mimicking human reasoning", the reality is more about "automated heuristics to recognize patterns in tons of samples"...

So, I will take the more accurate definition of AI, it is to say that I assume that an AI program can recognize patterns that might escape to the understanding of human being and that they can self-train to achieve this.

Now, speaking about disassembly, the core problems are definitely not about recognizing patterns but rebuilding all the execution paths that one can find in a binary. So, an AI algorithm is simply not interesting to choose if an execution path is more valuable than another because they are all interesting (as the disassembler need to exhaustively find all the paths).

Finally, speaking about decompilation, this is the field where you may want to use AI to recognize some patterns such as:

• What compiler or what language has been used;
• Guess if this code is similar to another sample that is in your data-base of malware;
• Give a first guess of the architecture of the software (complex data-structures (arrays, records), functions, objects, modules, packages, ...) even after an obfuscation pass;
• Recognize the most common functions and what they may be used for;
• ... probably many other guesses that can be automatized ...

But, the point here is that we are just all stuck at the disassembly problem (which is a quite strong problem!!!) and to go further we first need to have a reasonably reliable method/algorithm/tool in order to reach the next level (decompilation).

Though, a few people have worked in this direction, you may look at (this list is far from being complete, I search only 10mn on Google):

• Machine Learning-Assisted Binary Code Analysis, by Nathan Rosenblum, Xiaojin Zhu, Barton Miller and Karen Hunt, NIPS 2007.
• Deep Learning on Disassembly Data (Full talk), by Andrew Davis and Matt Wolff, BlackHat 2015.
• Using Recurrent Neural Networks for Decompilation, by Deborah S. Katz, Jason Ruchti and Eric Schulte, SANER 2018
• Evolving a Decompiler, by Matt Noonan, 2018;

Answer 2

Although I initially voted to close this question as primarily opinion-based, given both answers side with the same general answer ("No!") I'll answer as well. I just love being the devil's advocate.

A general stance

This question is very difficult to answer. As someone who came from the security field and had worked on quite a few machine learning related projects in the past, I know it's really hard to make an honest projection on whether it's possible to solve problems in our domain using machine learning. Often times ML/AI research is a lot about trial and error.

AI disassembler

Although the operation of disassembling a sequence of bytes is quite straight-forward, there are quite a few problems that are solved nowadays with somewhat clever heuristics. Those problems include:

1. Identifying function boundaries.
2. Classifying code VS data.
3. Inferring structures and higher-level constructs from assembly.

and more.

As time went on, some of those problems have been improved mostly by creating and improving manually written heuristics and logical guidelines for disassemblers. Mostly based on human experience and incorrect results seen in the past.

However, it is still quite common to see errors in fully automated disassemblers and as evidence to that we can see the focus given by disassemblers to the reverse engineer to manually correct and adjust the automated analyses. One might argue it's possible that ML/AI approaches could create similar or better heuristics and more complex / accurate rules.

AI decompiler

Decompilation is generally seen as more difficult to accomplish even given perfect disassembly. It relies heavily on inferring higher-level constructs such as structures, objects, classes and inheritance. It requires more accurate data-type recognition and more demanding understanding and reasoning of abstract objects. Even humans find those harder to manually recognize, not to mention using automated rules and heuristics. Therefore, decompilation may prove a more fertile ground for such improvements.

A word about the buzzword

Nowadays, when startups rise and fall based on using the right buzzword in their marketing it is easy to get drawn into using such buzzwords instead of accurate descriptions and forcing the solution on the problem instead of finding the best solution for the problem at hand.

Answer 3

No!

Reason is that AI is dumb, much more than you think it is. All it can do is matching patterns to predetermine output. And here are two main problem to use it to decompile code or to recreate "readable" source code:

• This matching isn't perfect. For example if you have mov, eax 0x10 it could be interpreted as i=8; as AI didn't have this particulate training set, but it has mov, eax 0x8.

• Other big problem is what to feed AI. In translation tools are used sentences, in image recognition are used features from whole images. What to use in code? Few op codes? How many? where to start?

In conclusion AI (at this stage and near future) is mostly useless for this purpose.

Answer 4

I want to say yes, but I have to say NO. The thing is that computers grab our programs that we write and optimize them in a way that often will make little or no sense to humans. You'll see things being multiplied and arrays being worked with that you can understand what is happening , but it is very unhuman to work with data in this kind of way. Computers and humans think very differently.

I think could be possible for an AI to do the following:

• Grab the assembly and convert and either make it clearer for humans to understand.
• Make a near useable form of assembly. (Maybe.... Because we would be keeping the computer's understanding of the relative data intact. )

Take this simple example:

MOV v7, DWORD PTR [v7 + 0x8]

would convert to

v7 = *(_DWORD *)(v7 + 8);

then to

v7 = *(v7 + 8)

And this might be really something completely different in source.

where as you might see something like

mov eax, dword ptr ds: [ESI*deadbeef+0b0] 

and the computer will think

int a = somefoovar[v5].someint

or

int a = 1234;

Both are kind of correct... If that array doesn't have changing data in it.

but a computer might look at this as a single variable, when it could be something temporary. Also I have noticed that in my own decompilation work, that you'll wind up with a lot more static variables that are when you originally start.

I think some other problems might be that the program might not ever hit certain parts of a function and might not understand all event paths.

Personally, I would like to see AI what AI can come up with . Maybe it would be nice to have a pseudo translator of code. But I know that I will have to clean up after it and correct it's understandings.