I'm in the middle of reverse engineering SafeNet (formerly Rainbow) iKey 1000 USB dongle protocol, in order to create Linux driver for it. I'm using Windows as guest in virtual machine on Linux host and Wireshark to examine USB packets. The protocol is quite simple, however I'm stuck with challenge-response MD5 authentication (ikey_MD5_CHAP).
The challenge is sent do the dongle in plain text then device calculates MD5 of challenge and key stored in device and sends it back to host. However, MD5 value transmitted back in control packet seems to be obfuscated for some reason.
The example value send via USB is:
09 77 e6 be 59 44 22 bc ef 60 47 da 4f bd be 80
when value returned by ikey_MD5_CHAP function by original software is
0a 9a 2b 26 e2 ef ef ab ed 3d 02 73 c5 11 f1 90
My goal is to discover the transformation function. There's great possibility that it consists of some XORs and additions since something similar was used in security officer password obfuscation as described in http://www.securityfocus.com/advisories/2440. I noticed that first byte is always incremented by one in all hashes that I had generated.
Since I'm reverse engineering newbie I'm asking for directions which should I follow to discover the transformation function. I can generate as many hashes as needed.
09 77 ...string to the USB dongle, which makes it respond with0a 9a ...? In that case, do what's being described in the CVE .. send a bunch of00bytes. Or does the dongle send09 77 ..., but the original windows software translates this to0a 9a ...? Then, disassemble the software. Or is it something else? Please clarify - what do you send? What does the dongle send? What do you expect? What does the original windows software transform?09 77...) which is then translated by the original software to clear one (0a 9a...). I need to discover the transformation function.