I am having a binary which takes an input from user and checks every bit of our character using the check_value function which does some stuff. The disassembly of the interested function is:

.text:000000000005620B loc_5620B:                              ; CODE     XREF: main+A1j
.text:000000000005620B                 movzx   edi, byte ptr [rbx+rdx]
.text:000000000005620F                 mov     rsi, [r8+rdx*8]
.text:0000000000056213                 call    check_value
.text:0000000000056218                 inc     rdx
.text:000000000005621B                 and     ecx, eax
.text:000000000005621D                 cmp     rdx, 3Bh   <----**This condition**
.text:0000000000056221                 jnz     short loc_5620B
.text:0000000000056223                 test    cl, cl
.text:0000000000056225                 lea     rdi, aYouDidnTGetItM ; "You didn't get it, much sadness :("
.text:000000000005622C                 jz      short loc_5623E
.text:000000000005622E                 lea     rdi, aYouGotItCorrec ; "You got it! correct! awesome!"
.text:0000000000056235                 call    _puts
.text:000000000005623A                 xor     eax, eax
.text:000000000005623C                 jmp     short loc_56248

What I understand is the cmp instruction is where the execution is getting diverted and I want to do a straight bruteforce. How can i do using either a gdb script or angr framework?

2 Answers 2


By the context I think this is the rabbithole problem from CSAW ctf finals. The correct solution was not to bruteforce although you'd be tempted to as it was trivial to brute-force. The check_value function was called for each char in the input which was supposed to be 0x3b chars long.

for(i=0; i<59; ++i){
    flag &= check_value(input[i], roots[i])

where roots is an array for roots to binary trees which were traversed in check_value. I solved it onsite and if you want to look at how I did it check here.

  • You can also checkout other writeups at ctftime.org/task/4915
    – sudhackar
    Commented Dec 7, 2017 at 17:02
  • @sudhacker, yes I saw your writeup which included SMT solver but I thought a bruteforce would work in this case.
    – ashish
    Commented Dec 7, 2017 at 17:08
  • yea check out this one anee.me/rabbithole-csaw-ctf-2017-finals-f7d70f3726f3. He bruteforced it.
    – sudhackar
    Commented Dec 7, 2017 at 17:09
  • yes, I tried that but sadly it didn't work!
    – ashish
    Commented Dec 7, 2017 at 17:11
  • As I mentioned flag is the and of all calls to check_value. Every call must return 1(true) in order for you to have the correct flag. You can set a bp there and iterate byte by byte so that each call returns 1.
    – sudhackar
    Commented Dec 7, 2017 at 17:15

Brute forcing is often not the way to go unless you know something about your input that would limit the space of what it needs to check considerably. I think you're best bet here it to understand what is going on in the machine code especially given how straightforward the disassembly looks.

Let's break it down.

movzx   edi, byte ptr [rbx+rdx]
mov     rsi, [r8+rdx*8]
call    check_value

Here we can see the program is moving two values into the rsi and edi registers. They are both indexed with rdx so rdx is probably an index counter. edi is loaded with a byte so it's probably part of the password/value that is being checked. Therefore, rbx is a pointer to the password/value itself.

inc     rdx
and     ecx, eax
cmp     rdx, 3Bh
jnz     (5 instructions above)

Increments the index counter, looks at the return value (eax) from check_value and clears register ecx if the return value is zero. Finally, it compares the counter to 0x3b (59 decimal) and repeats everything above it not equal to it. From this, we can gather that our secret password/value is 59 bytes and that check_value must likely return 1 (true) for every byte if the value is correct.

test    cl, cl
lea     rdi, aYouDidnTGetItM
jz      short loc_5623E
lea     rdi, aYouGotItCorrec
call    _puts

Here we can see we are getting either one of two messages put depending on the value of cl. If ecx is zero from the loop above, cl will be zero and the jz jump will be taken leaving us with the message "You didn't get it, much sadness :(" but if ecx is non-zero, we get "You got it! correct! awesome!" which is then displayed on the console with the puts function.

So, from just breaking this down a bit we can see exactly what this function is doing.

Extrapolating a bit from missing disassembly in your listing above (what sets ecx before your listing starts and what prints out the didn't-get-it-message after your listing ends), the C code for this might looks something like

bool passwordOkay = True;
for (index = 0; index != 59; ++index) {
    byte = password[index]
    secondVal = table[index]
    passwordOkay &= check_value(byte, secondVal);
if (passwordOkay)
    printf("You got it! correct! awesome!");
    printf("You didn't get it, much sadness :(");

You should apply the same technique within the check_value function to reverse exactly hot to defeat this checker.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.