Retrieving encryption/encoding logic from dll

Question

I am trying to retrieve a decrypting/deconing logic from a program. Unfortunately I have virtually no experiencing in reverse engineering.

I am building an alternative controller application for a multimedia device. The original controller is available for Windows (which I target) as well as for mac, ios and android. Most of the communication between the device and the controller application is similar to the UPnP protocol, therefor teransmitted in XML and readable in cleartext. There is one instance however where the device sends some sort of token to the controller (which the controller uses subsequently to authenticate against other services) and this transmission seems to be encoded/encrypted/obfuscated (referred as "encoded" in the text). This encoded form is transmitted in an XML field of a normal UPnP response.

I have made the following observations while monitoring several transmissions of the encoded form:

• It always starts with 2: and is followed by what seems to be a base64 encoded string
• When transmitted multiple times the sting changes even though the content supposedly did not change (this is under the assumption that no other variables like a timestamp are included before encoding). So some kind of salt could be present.
• The length of the string remains the same if the data is not changed
• If removing all tokens from the controller there is still some encoded data transmitted in this field. This leads me to thinking that either padding takes place, or (more likely) the encoded string decodes to some serialized from of data structure (maybe even XML).
• The token that is used for authentication (so the decoded form) does remain the same even though the encoded version differs.

The application itself is written in c# but all the security related stuff (including decoding/handling the token) seems to be called using pINVOKE from a DLL which was not written in .net.

I ran signsrch on the dll where I suspect the encoding to take place however a lot of possible hits are returned (30+). When taking a look at the dll it seems that the openssl library is compiled into it so there might be a lot of false positives. The DLL also handles HTTPS communication so openssl is not necessarily related to the encoding method I am looking for.

I've tried to look into ida pro and windbg however what I read from tutorials this mainly revolves around setting breakpoints before clicking on a button that activates e.g. a check routine. However in my case the controller application receives this data once on application start (with dozens of other UPnP packets) and my guess is that decoding takes place on receiving the packet so there is no possibility to set an breakpoint to e.g. a button. What I was able to figure out however is that the decoded tokens are present in memory (however offsets differ each time the application is started). The encoded version does not seem to be present in memory at runtime.

Some examples: Encoded String when no token is configured:

2:nrlrLkCeoSOK34YSDZUKzbXs6olm9YSE4jLFnXONMJO5XhjfwCsPpymjHEXBhuSiRUAYpghW1NHj7M2zIgzcbw==

Three strings representing (supposedly) the same data:

2:PITkxbk12KlLvsPtOV8ptZbqfMcK0EmSA/XOejbBHs8+ffKNzwYSoz6oGdL5wiudTWJri4ZG/CwB2/Pfh91ULuqBYWaLaW7YCzEzRb53NS+qarNiUAU0Kn7TCuhPzougX91sKFOP3BTWdUu6BQBP7IlWIZ1odp4ilU8RdEUhPvkOVyH9OBQwvpYXPbqca/OLGjNFPvomS5yaeOjtbpDuVd8mfvBycVHB8rULDvkJcXOS/qQWHOt8iYYZXHBXBD9v4h3Bs+o09sWsXxIh+0q4qz+TKY/gYRnogenJ9vN83lkE6FTgtHns/6zxlCt4xaFINmShdovvYlS/sdZ+aKgcGoq8zFjWJAbc2WQQwdxPTLZsCUmJSjH6jp1t6pCv7t4tBMaSpCTvAoRrXElNQIW1LOe/9X6Cu5Ggjp17Whwe9+7DuWz6HxPEpq/TIVLIV7kJQRj6GcD7/sWPakLLXxiN1Bf5yfJVvX3wA67vwXwigO5v5d5JzKPrjULskeCLUk//61gPulxFguf63cn6NN1zXpDBOf3sCxwzquKIobE8s+F2mJ0pzWCNwYSg3IpvAUzH2fqLvsjqO6DdNr6AVZr77vPG9/dQ2+sFesFHMFQ859eVwNUSvYkxQoqFukxJOXfY+muf1AYqczvCA0K/uYDnKAj24oB0Wgw9YvAS8YHO9YI=

2:XfaIk9XLov9k8vHKXXZ9NFpUvNSk8QsEjI73E+txW6wg+kSr2hWWljLU5OOmQBcN+ACRKF4f4paJBxlf9AGiFnWyrNKT4nguM/4iBsHsiIADlMavjYncsNntClcxPh+Me2l1W/uQzqG83SlHcM525P5wfpH45zC7/hFjmWKPGIQ/k/LAsJy6gZlrmOZ0B2JJ0Cw+yUOzTD4EUo092EPRf7hq7aIrs0dLfPtT2D6LMKwmKAfW2fhllV/jJ33dAVn6Dy/PctFErGtvXljhnuI8r5qpMyIOZ5SsoFE/tnk+kOqWYVg5/Djbl/fpV2p70B9vxzpVkko8pfIJ3UmYsqu5WSlaX441Qs/gY8nzjWmJw8+8LzxL+P2YgYo5W+u6a/oS4R+sZ3FD0q3KQvmVNjklgdGipTOfsv9amP6M6ifAHPdgB5kj9ivOtufaiFKR9GqS9/g4vjYQqofU3Yk2nB6xMi2W1eA2Mk+BJAjfsTr1Yo4Njoa0+XQfAMfqpJe9Zka2NSLu2noDDEdRTqASJi6IiNxbsbsVGHhHFEAEojwTQlrk6xUtOt5RjTA6XovdgQoGywjU+FixEbEEQA0TbNE2KsFhmKXIauo3yQaZ9EKDMJjOMz53UK/EFEZu6L7yy7ijnQB13rSw+obIbFXr/J/vxc97EsRXm7zSUIQoEq9qwKE=

2:9xVbeGOq4XDN26aIYcbNVYDAVQNgVseVvipidY71ZzJAKYlYdcDd46uIFQdfla5pXqupZABtl4Njg+wC8lxmxx2EFR1ZwBYr+ZbvswCQp9YKKMAhI3Ly12j3XNrtIAfl+EDDgz5woHNLXLfVthFel4WSe3mGHMa0S8ZcXRyRyLJHx7lIXZ4G68by2cLTYTldAWuYYl7xhmIJNLgMA8iI63FBGYiw/aXd/30KkLPoUP7As1R/GfItnDfp0pG5l63DtBFSo3mnDRVrdDIsYALRpsBLhhvx4+0CjlyMaK511pM0+IxmtcE6L0L6QhcztYzWcjnbjkLi7r9dmO5ABEfq8/8+wpnxERXwiwRTsYPy2/+rYQlLK0hkeGiqacbnjQQWyVRVzaE47oAhC9ln+ws5go/v+reDHhmpf/9vj9RO7EzFq9LVZRxbxy7C5QZzHEqncSdR+TzOrs0MkRO3pe+e2Q5P68kgTYVO6K8XpvLU9ClfDyIOCpwNufRz/fEVl2ry5UrjS5d5k39+UheQizu9nu+Y+4xC42Wd+y1E7WsZYQaLgNL6EQP1w8yb3+PwdvgOxgId7IMag81/4+mjO9x/N1Zoq405nSXodirZ5RiWIAl4+uHARqwDnX1lZQo7Fo0pMs8ZwOtGiqA8/WVbMKqigBOEW6m2/2K2nI3cTYKqmVg=

I'd love to hear suggestions how to approach this problem to get more insight.

Answer 1

Cant comment, so an answer it is :)

-- If you know what dll it is you need to look at and its written in .net, IDA will probably not be the best option (from personal experience).

If however you are still looking for a dll which you need to decompile, you might want to give APImon and procmon a shot to find what exactly needs decompiling.

I would suggest try using a .net decompiler, for example dotPeek or any other. You can generally read code as is from any .net program (not obfuscated / encrypted) If a dll is obfuscated, you can try deobfuscating it with de4dot or dump from memory and analyze that instead.

Above strings are definitely base64 encoded, so you are probably going to be looking for object.binary.base64

Answer 2

Have a look at the plugin for IDA Pro called IDA Scope. It has a crypto-identification facility that I think will help you.