Making this an answer because the comment history got too long.
The source code of the class in question seems there's one function
void GenerateKey(BigTwistedEdwards *math, IRandom *prng, Leg *key) that creates the key in the
Leg class. I'd assume this is created with key=new Leg somewhere, so it won't be one single address. (In fact,
Leg might be a
uint32 or a
Leg *key is a pointer to an array of these integral values).
Maybe, if you find the
GenerateKey function in the binary, you can add a few calls to
CloseFile to the end of it to get the key, and maybe the executable detects if a debugger is present, but doesn't detect if the executable itself is modified.
It seems that
KeyAgreement.cpp is the file that holds the
GenerateKey function. It generates a key until the
do .. while loop finds the key acceptable, so the end of that
do .. while loop is where you want to place your breakpoint (if you can debug the executable) or add a call to a function that writes the key to a file.
Now, we need to find the
GenerateKey function in the executable, or in one of its DLLs. You can use any tool that lists DLL exports in the hope that one of your DLLs exports
KeyAgreementCommon::GenerateKey. If you're lucky, you'll find it there, and you're done. Dependeny Walker is a good tool for this.
Assuming the function is compiled into the main .exe, or in a .dll but not exported, it's a bit harder. Fortunately,
KeyAgreement.cpp holds a bunch of nice character arrays with very distinct signatures, which should be unique and easy to find. You could try one of these:
- add the
Q_XXX arrays to signsrch and run it over all .exes and .dlls
- load your .exe and the .dlls into a hex editor individually, and run a byte search for the
Q_XXX arrays over them, until you find the correct one.
Next, you want to know where these bytes are used. The easiest way to do this is load the file you found to be the correct one into Ida Pro, find your character array there, and check the XREFs. There should be only one of them, that comes from
KeyAgreementCommon::InstantiateMath. The function after this should be
KeyAgreementCommon::GenerateKey, if your developers didn't change the source code.
If they did change the source code, maybe they shuffled around the functions a bit. (And maybe they decided they didn't want to use anything but the 384 bit version, so they threw away the 256 bit and 512 bit arrays. This is why i said search for all 3 of them.) Now we want to find the
GenerateKey function when we only know the
Fortunately, both of them are declared public in the .hpp file. Which means they should occur in the class vtable somewhere. Check Ida for a data x-reference to the
InstatiateMath function. This should be the vtable entry. The same vtable should hold the addresses of the
GenerateKey function, so check the functions that are referenced next to the