I'm trying to reverse a Linux module. In the ioctl routine, I have the following sequence of instructions (generated by objdump -M intel -d test.ko):

 3f9:   74 3b                   je     436 <device_ioctl+0x46>
 436:   e8 00 00 00 00          call   43b <device_ioctl+0x4b>
 43b:   48 98                   cdqe   
 43d:   c3                      ret

I'm not sure to understand what happens here after the jump is done.

First, we have a call instruction. The opcode is E8 00 so it is a near call to the next instruction. The EIP register (which contain the address of the next instruction) is also pushed on the stack.

Then, we continue with the cdqe instruction. It extends the sign of the EAX register into RAX. In other words, it does nothing. Just to mention, EAX/RAX register has not been modified from the beginning of the routine.

Finally, the ret instruction is executed. It pops the stack into EIP which corresponds to the previous instruction and we continue the execution of the routine from that point.

Therefore, cdqe is executed again (to do nothing) and the ret instruction is called again. This time, we leave the ioctl routine and return to the callee.

Am I right saying this does nothing? If so, what is the point of that code?

Here is the full routine:

00000000000003f0 <device_ioctl>:
 3f0:   48 89 d7                mov    rdi,rdx
 3f3:   81 fe 03 b0 fe ca       cmp    esi,0xcafeb003
 3f9:   74 3b                   je     436 <device_ioctl+0x46>
 3fb:   77 18                   ja     415 <device_ioctl+0x25>
 3fd:   81 fe 01 b0 fe ca       cmp    esi,0xcafeb001
 403:   74 39                   je     43e <device_ioctl+0x4e>
 405:   81 fe 02 b0 fe ca       cmp    esi,0xcafeb002
 40b:   75 39                   jne    446 <device_ioctl+0x56>
 40d:   e8 00 00 00 00          call   412 <device_ioctl+0x22>
 412:   48 98                   cdqe   
 414:   c3                      ret    
 415:   48 c7 c0 00 00 00 00    mov    rax,0x0
 41c:   48 c7 c2 ff ff ff ff    mov    rdx,0xffffffffffffffff
 423:   48 2d 00 00 00 00       sub    rax,0x0
 429:   81 fe 04 b0 fe ca       cmp    esi,0xcafeb004
 42f:   89 c0                   mov    eax,eax
 431:   48 0f 45 c2             cmovne rax,rdx
 435:   c3                      ret    
 436:   e8 00 00 00 00          call   43b <device_ioctl+0x4b>
 43b:   48 98                   cdqe   
 43d:   c3                      ret    
 43e:   e8 00 00 00 00          call   443 <device_ioctl+0x53>
 443:   48 98                   cdqe   
 445:   c3                      ret    
 446:   48 83 c8 ff             or     rax,0xffffffffffffffff
 44a:   c3                      ret    

This is from the nightclub challenge of the pbctf 2021.


cdqe does this RAX ← sign-extend of EAX.
applicable only in x64
shown below is a demo of x86 cdq which instead of using rax uses eax,edx combo edx gets the sign extension in x86

for cdqe register rax will become 0xffffffff<eax> or 0x00000000<eax>

enter image description here

maybe ctf is checking the sign

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.