I'm reversing a C++ program using IDA PRO to examine how the some data is stored, encrypted and decrypted.
I've managed to identify the data is encrypted with AES CBC 16 block and 256 bit key. I have tested this as I have found the generated key in the program and managed to decrypt the data using online tools.
The problem I have is working out how the AES 256 bit key is generated. I have identified the code but is several hundered lines long. I "think" its a cut n copy of one of the hash functions from OpenSSL as I see references to that in other areas of the code but I cant be 100% sure.
There isn't much to go on in the code except a few magic numbers like 0x2AAAAAAB and that 3 DWORDS are pushed to the stack before the sub is called.
I've downloaded the OpenSSL code but cant find anything yet. I've also used tools like Detect it Easy and a Plugin for IDA, which finds some crypto signatures which I BP, but they are never hit when stepping over the keygen function.
Can anyone recommend anything what can help which this ? My ASM skills are no where near good enough to Single Step several hundreds of instructions.
EDIT: I used the Hex Rays decomplier for the specific func and it churned out only a few lines of code !?!
I've put it on paste bin at:
Can anyone explain what it does ? However I'm not even sure this is the correct code as no matter where I press F5 to generate the Pseudo Code, it gives me the same output.
EDIT: here is the paste of the full code