You did not mention what the format of the binary is. If the decompiler you choose to extend does not support the format of the binary you want to decompile then you will need to either implement support for that format or find a work-around.
There are a number of open-source decompiler projects under active development that you can choose from. Of course, there are also plenty of dead compiler projects, but they will not be discussed.
The projects discussed here are
- In the r2con2018 presentation "How to not write a decompiler" (video, slides), an overview of how the radare2 plugin
r2dec works is given, as well as some instructions and advice on how to extend it to new architectures (beginning at 17:59 of the presentation). If radare2 (a disassembler) does not support the architecture you are interested in, then you will need to implement this yourself.
reko - standalone toolchain implemented in C#
- See the HowTo in the wiki for instructions on how to proceed with extending it to a new architecture
- RetDec - uses the Capstone disassembly framework and is based on LLVM
- An overview of how it works is provided here: (slides, video). Discussion of disassembly and translation to IR begins at around 17:08 (slide 25). Extending it to an unsupported architecture probably will involve adding support to Capstone for disassembly as well as the asm -> IR translation in RetDec.
This is not meant to be an exhaustive list, of course. There are other options as well, such as radeco and snowman, but these do not seem to be very well documented.
If it is decided that decompilation is not required and disassembly will suffice, a nice guide to adding a new processor module to IDA is given in the following articles: