I'm in the middle of reverse engineering SafeNet (formerly Rainbow) iKey 1000 USB dongle protocol, in order to create Linux driver for it. I'm using Windows as guest in virtual machine on Linux host and Wireshark to examine USB packets. The protocol is quite simple, however I'm stuck with challenge-response MD5 authentication (
The challenge is sent do the dongle in plain text then device calculates MD5 of challenge and key stored in device and sends it back to host. However, MD5 value transmitted back in control packet seems to be obfuscated for some reason.
The example value send via USB is:
09 77 e6 be 59 44 22 bc ef 60 47 da 4f bd be 80
when value returned by
ikey_MD5_CHAP function by original software is
0a 9a 2b 26 e2 ef ef ab ed 3d 02 73 c5 11 f1 90
My goal is to discover the transformation function. There's great possibility that it consists of some XORs and additions since something similar was used in security officer password obfuscation as described in http://www.securityfocus.com/advisories/2440. I noticed that first byte is always incremented by one in all hashes that I had generated.
Since I'm reverse engineering newbie I'm asking for directions which should I follow to discover the transformation function. I can generate as many hashes as needed.