I've recently took it upon myself to research more about reverse engineering, I'm currently testing my intermediate skills on various CTF hosting sites. Interestingly enough I have come across a challenge on this particular site:
Now the challenge I'm focused on at this moment in time is
So far what I've gathered about this challenge is that it has some form of program obfuscation, in this case the obfuscation I've identified is a VM (Virtual Machine) which fetches and decodes a large set of opcodes. The opcodes are stored within the same function as the entry point of the program.
Now I have identified each of the custom VM instructions and have created a very short IDA Python script to extract the bytecode. It isn't ideal but it does what it is suppose to do.
dism_addr = list(FuncItems(0x1b4c)) for addr in dism_addr: if "mov byte ptr [rax]," in GetDisasm(addr): print(GetDisasm(addr))
After creating a disassembler for the challenge, I'm curious if there is a better way to solve this challenge rather than spending a couple of hours (2/3) reverse engineering. I've read about Symbolic Execution and my question is:
How would a reverse engineer solve a challenge like this using Symbolic Execution?
Please refrain from posting flags, I just want a thorough explanation on what I could do to solve this more efficiently and quickly.
Any help is appreciated, thank you for taking the time to read my question!