I am trying to reverse engineer an executable that uses a lot of control flow flattening obfuscation (i.e, lots of subroutines that end with something like jmp esi, jmp eax) IDA Pro isn't able to resolve where the jump can branch off to and I'm not really sure what to do here.
I made a plugin for x64dbg ( https://github.com/JeremyWildsmith/x64dbg_scripts/blob/master/traceAreas.py ) that locates and monitors the branching instructions in a provided list of subroutines and generates graphml diagrams that I am reading (i.e example like this: https://i.stack.imgur.com/4StsO.png ) that I am viewing and analyzing in yEd live.
But it generates a lot of data. I know there is a vm hanging around somewhere in there and I am having trouble finding it just because of all the obfuscation. The collected control-flow data is here (warning, big text file, 45kb) https://github.com/JeremyWildsmith/x64dbg_scripts/blob/master/flow.graphml
When I open it in yEd Live it looks like there are multiple virtual machines or something? Yeah not really sure where to go with this, if anyone has any advice that would be awesome.
If I look at the control flow in a radial diagram ( https://i.stack.imgur.com/YrHn4.jpg ) I think I can identify the center as some sort of dispatcher or interpreter for the bytecode instructions? But what throws me off is how much branching goes on later on through the execution.