What are the qualitative differences in reverse engineering x64/Win64 code versus x32/Win32 code?

Question

Many professional reverse engineers spend their days looking at 32-bit code compiled for Windows, and familiarity breeds proficiency. What are the high-level differences between reverse engineering 64-bit Windows programs versus 32-bit ones?

I am talking about things that will be staring me in the face all the time, as opposed to, say, slightly differing behavior of APIs. For instance,

• Do 64-bit compilers employ particular optimizations that are infrequently seen in a 32-bit context?
• Will I see instructions in 64-bit programs that are different from, or non-existent in, the 32-bit instruction set?
• Is there anything in particular that I need to know about the tool support on x64 versus x86?

Answer 1

One of the biggest differences between x86 and x86_64 is the introduction of RIP-relative addressing. Similar to 32-bit ARM, data can now be (easily) referenced at an offset from the current RIP value.

For example, here are the first few instructions of __libc_csu_init() in a x86 program:

08048420 <__libc_csu_init>:
 8048420:       55                      push   %ebp
 8048421:       57                      push   %edi
 8048422:       31 ff                   xor    %edi,%edi
 8048424:       56                      push   %esi
 8048425:       53                      push   %ebx
 8048426:       e8 f9 fe ff ff          call   8048324 <__x86.get_pc_thunk.bx>
 804842b:       81 c3 79 12 00 00       add    $0x1279,%ebx
 8048431:       83 ec 1c                sub    $0x1c,%esp
 8048434:       8b 6c 24 30             mov    0x30(%esp),%ebp
 8048438:       8d b3 0c ff ff ff       lea    -0xf4(%ebx),%esi

And here it is on x86_64 (note 0x40050a and 0x400511):

0000000000400500 <__libc_csu_init>:
  400500:       48 89 6c 24 d8          mov    %rbp,-0x28(%rsp)
  400505:       4c 89 64 24 e0          mov    %r12,-0x20(%rsp)
  40050a:       48 8d 2d 97 01 20 00    lea    0x200197(%rip),%rbp        # 6006a8 <__init_array_end>
  400511:       4c 8d 25 88 01 20 00    lea    0x200188(%rip),%r12        # 6006a0 <__frame_dummy_init_array_entry>
  400518:       48 89 5c 24 d0          mov    %rbx,-0x30(%rsp)
  40051d:       4c 89 6c 24 e8          mov    %r13,-0x18(%rsp)
  400522:       4c 89 74 24 f0          mov    %r14,-0x10(%rsp)
  400527:       4c 89 7c 24 f8          mov    %r15,-0x8(%rsp)

You can find more information about this convention here: http://www.codegurus.be/codegurus/programming/riprelativeaddressing_en.htm

Answer 2

One of the big things that will be different is the calling convention - on 64-bit some parameters are passed in registers; see http://en.wikipedia.org/wiki/X86_calling_conventions and the references

Other obvious things include the larger registers, more SSE registers, and 64-bit arithmetic (mov QWORD / movq et al.). Beyond what you would expect, things are actually fairly similar. See http://en.wikipedia.org/wiki/X86-64#Architectural_features for an overview of the large differences - most of the other new features are of more importance to kernelspace code rather than userspace code. Beyond logical extensions of 32-bit instructions to 64-bit instructions, the instruction set remains fairly static.

Answer 3

Regarding tool support, Ollydbg (and Immunity Debugger) does not support x64. Windbg is probably the best free alternative.