I'm working on reversing some assembly code, and have worked it up into some pseudo code to start off with a reverse attempt. The function performs a kind of encryption on the data passed in, and I'm trying to decrypt it. However, I'm not quite sure where to begin with the reversal itself. Below is the algorithm I'm trying to reverse.

Question: Since the encoded string is stored in rev3, wouldn't the string data stored there be obliterated when the AND instruction runs? I'm very new at this, but it seems like that wouldn't be reversible. The AND combined with shifts has me really confused. I would be grateful for an approach to reversing this.

string = [102, 111, 111] // the string "foo"
for (i = 0; i < len(string); i++){

  rev1 = rev2 = rev3 = string[i]

  rev3 = rev3 & 0x30 // obliterate here?
  rev1 = rev1 >> 4

  rev3 = rev3 ^ rev1
  rev1 = rev2

  rev1 = rev1 & 1
  rev2 = rev2 << 1

  rev1 = rev1 ^ rev2
  rev3 = rev3 >> 2

  rev1 = rev1 << 2
  rev3 = rev3 ^ rev1
  rev3 = rev3 & 0xff

  coded[i] = rev3


1 Answer 1


The simplest solution

Since the function you provided operates on characters, there are only 256 possible inputs, so you may run a simple program to check return values for all chars from 0 to 255 to get :

#include <stdio.h>

int main()
    int rev1, rev2, rev3;
    for (int i = 0; i < 256; i++)
        rev1 = rev2 = rev3 = i;
        rev3 &= 0x30;
        rev1 >>= 4;
        rev3 ^= rev1;
        rev1 = rev2;
        rev1 &= 1;
        rev2 <<= 1;
        rev1 ^= rev2;
        rev3 >>= 2;
        rev1 <<= 2;
        rev3 ^= rev1;
        rev3 &= 0xff;
        printf("%d: %d\n", i, rev3);

You will get all possible values along with arguments, so you may just invert the output of the program above and you will get decrypting function.

  • Thank you for the help. I wrote one line incorrectly. rev3 &= rev1 should be rev3 = rev3 ^ rev1 I am testing it out now with this replacement.
    – TyManning
    Jul 7, 2019 at 15:13
  • Well, that changes a lot. :) Give me some time - I will edit my answer.
    – bart1e
    Jul 7, 2019 at 15:16
  • Thank you very much. I feel silly to not have seen such a simple solution. :)
    – TyManning
    Jul 7, 2019 at 15:20
  • 1
    You are welcome. Sometimes the simplest solution is hardest to see. And, by the way, ANDs and SHIFTS are invertible here because the information about argument is stored in rev1 and rev2 as well. If you only had rev3 and perform one of these operations, it wouldn't be invertible.
    – bart1e
    Jul 7, 2019 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.