I'm working on reversing some assembly code, and have worked it up into some pseudo code to start off with a reverse attempt. The function performs a kind of encryption on the data passed in, and I'm trying to decrypt it. However, I'm not quite sure where to begin with the reversal itself. Below is the algorithm I'm trying to reverse.

Question: Since the encoded string is stored in rev3, wouldn't the string data stored there be obliterated when the AND instruction runs? I'm very new at this, but it seems like that wouldn't be reversible. The AND combined with shifts has me really confused. I would be grateful for an approach to reversing this.

string = [102, 111, 111] // the string "foo"
for (i = 0; i < len(string); i++){

  rev1 = rev2 = rev3 = string[i]

  rev3 = rev3 & 0x30 // obliterate here?
  rev1 = rev1 >> 4

  rev3 = rev3 ^ rev1
  rev1 = rev2

  rev1 = rev1 & 1
  rev2 = rev2 << 1

  rev1 = rev1 ^ rev2
  rev3 = rev3 >> 2

  rev1 = rev1 << 2
  rev3 = rev3 ^ rev1
  rev3 = rev3 & 0xff

  coded[i] = rev3


The simplest solution

Since the function you provided operates on characters, there are only 256 possible inputs, so you may run a simple program to check return values for all chars from 0 to 255 to get :

#include <stdio.h>

int main()
    int rev1, rev2, rev3;
    for (int i = 0; i < 256; i++)
        rev1 = rev2 = rev3 = i;
        rev3 &= 0x30;
        rev1 >>= 4;
        rev3 ^= rev1;
        rev1 = rev2;
        rev1 &= 1;
        rev2 <<= 1;
        rev1 ^= rev2;
        rev3 >>= 2;
        rev1 <<= 2;
        rev3 ^= rev1;
        rev3 &= 0xff;
        printf("%d: %d\n", i, rev3);

You will get all possible values along with arguments, so you may just invert the output of the program above and you will get decrypting function.

  • Thank you for the help. I wrote one line incorrectly. rev3 &= rev1 should be rev3 = rev3 ^ rev1 I am testing it out now with this replacement. – TyManning Jul 7 '19 at 15:13
  • Well, that changes a lot. :) Give me some time - I will edit my answer. – bart1e Jul 7 '19 at 15:16
  • Thank you very much. I feel silly to not have seen such a simple solution. :) – TyManning Jul 7 '19 at 15:20
  • 1
    You are welcome. Sometimes the simplest solution is hardest to see. And, by the way, ANDs and SHIFTS are invertible here because the information about argument is stored in rev1 and rev2 as well. If you only had rev3 and perform one of these operations, it wouldn't be invertible. – bart1e Jul 7 '19 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.