What's a decompiler that can be extended to new architecture, and how can I do it?

Question

I was reverse engineering a binary file. Details does not matter. The important part is that it is written in an architecture that is very unpopular, and there is no existing disassembler or decompiler that supports the architecture.

It's possible to write a disassembler without too much difficulty. Also, assume that I have the full documentation on how the machine code is executed.

However, writing a decompiler is much harder. So instead, I am thinking about using an existing decompiler and extend it to support my architecture.

Question:

• What is a decompiler that can be extended to new architecture?
• How can I extend the decompiler to support my architecture?

Answer 1

You did not mention what the format of the binary is. If the decompiler you choose to extend does not support the format of the binary you want to decompile then you will need to either implement support for that format or find a work-around.

There are a number of open-source decompiler projects under active development that you can choose from. Of course, there are also plenty of dead compiler projects, but they will not be discussed.

The projects discussed here are

1. r2dec - a plugin for radare2 written in JavaScript

   • In the r2con2018 presentation "How to not write a decompiler" (video, slides), an overview of how the radare2 plugin r2dec works is given, as well as some instructions and advice on how to extend it to new architectures (beginning at 17:59 of the presentation). If radare2 (a disassembler) does not support the architecture you are interested in, then you will need to implement this yourself.

2. reko - standalone toolchain implemented in C#

   • See the HowTo in the wiki for instructions on how to proceed with extending it to a new architecture
3. RetDec - uses the Capstone disassembly framework and is based on LLVM
   • An overview of how it works is provided here: (slides, video). Discussion of disassembly and translation to IR begins at around 17:08 (slide 25). Extending it to an unsupported architecture probably will involve adding support to Capstone for disassembly as well as the asm -> IR translation in RetDec.

This is not meant to be an exhaustive list, of course. There are other options as well, such as radeco and snowman, but these do not seem to be very well documented.

If it is decided that decompilation is not required and disassembly will suffice, a nice guide to adding a new processor module to IDA is given in the following articles:

• How we developed the NIOS II processor module for IDA Pro
• Flare-On 2018 - Challenge 12 - Subleq'n'RSSB

Answer 2

You can use Ghidra for this: https://ghidra-sre.org/

Creating new CPU descriptions is possible. There's nothing obvious in the way of documentation, but there are plenty of examples in the Ghidra/Processors folder. I managed to muddle through and create a new one for the 6502, even though I didn't quite understand everything (and still don't): https://github.com/tom-seddon/ghidra_6502

The decompiler appears to be driven entirely by these CPU descriptions, so I think all you need to do is make a new description, and you'll get C code, of a sort, out of it.

I'm not sure how feasible it is to turn a whole program into some code that you can recompile! - that's not something I need, so I haven't investigated that. But if you just need particular routines disentangling, it may suffice.