Cross debugging for ARM / MIPS ELF with QEMU/toolchain

Question

as i'm new about cross-debugging and cross-compilation i need some help because i feel so confused. I have a MIPS elf file, [myelf][1] .You can see bellow the output of file myelf:

myelf: ELF 32-bit LSB executable, MIPS, MIPS-I version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, BuildID[sha1]=0xc89c3571514c7ec1afc74a189a9c2d24e276ec4c, with unknown capability 0xf41 = 0x756e6700, with unknown capability 0x70100 = 0x1040000  stripped

I just want to run and debug the program. So i don't need a cross compiler right ? As i don't have MIPS hardware (i have an INTEL microprocessor), i need an emulator. I've chosen to use QEMU. According to this site, i downloaded the following kernel image and initrds:

debian_squeeze_mips_standard.qcow2

vmlinux-2.6.32-5-4kc-malta

Then i've run the specified command for a 32 bit (because the elf informations) MIPS system.

qemu-system-mips -M malta -kernel vmlinux-2.6.32-5-4kc-malta -hda debian_squeeze_mips_standard.qcow2 -append "root=/dev/sda1 console=tty0"

So far, i have the emulator running in one shell and the command uname -a gives me:

Linux debian-mips 2.6.32-5-4kc-malta #1 Tue Sep 24 00:02:22 UTC 2013 mips GNU/Linux

There are only the very basics commands/tools on the emulator. I've read that gdb can debug on a remote target (here the MIPS-emulator) from an host machine which is my x86_64. And to be honest i have no idea about what i should do now. I first tried to install gdb itself on the the qemu emulator.When i run gdb my elf i can see that gdb was automatically configured as mips-linux-gnu.

root@debian-mips:~# gdb myelf 
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mips-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/myelf...(no debugging symbols found)...done.

The info files gives me the right informations (i disassembled the elf with IDA so i can confirm).

(gdb) info files 
Symbols from "/root/myelf".
Local exec file:
    `/root/myelf', file type elf32-tradlittlemips.
    Entry point: 0x400670
    0x00400154 - 0x00400161 is .interp
    0x00400164 - 0x00400184 is .note.ABI-tag
    0x00400184 - 0x0040019c is .reginfo
    0x0040019c - 0x004001c0 is .note.gnu.build-id
    0x004001c0 - 0x00400298 is .dynamic
    0x00400298 - 0x0040033c is .hash
    0x0040033c - 0x0040049c is .dynsym
    0x0040049c - 0x0040057b is .dynstr
    0x0040057c - 0x004005a8 is .gnu.version
    0x004005a8 - 0x004005d8 is .gnu.version_r
    0x004005d8 - 0x00400668 is .init
    0x00400670 - 0x00400b00 is .text
    0x00400b00 - 0x00400ba0 is .MIPS.stubs
    0x00400ba0 - 0x00400bec is .fini
    0x00400bec - 0x00400c2c is .rodata
    0x00400c2c - 0x00400c30 is .eh_frame
    0x00410c30 - 0x00410c3c is .ctors
    0x00410c3c - 0x00410c44 is .dtors
    0x00410c44 - 0x00410c48 is .jcr
    0x00410c50 - 0x00410e00 is .data
    0x00410e00 - 0x00410e04 is .rld_map
    0x00410e10 - 0x00410e6c is .got
    0x00410e6c - 0x00410e70 is .sdata
    0x00410e70 - 0x00410e80 is .bss

But when i want to run the program nothing is happening:

(gdb) r
Starting program: /root/myelf 

I waited about 10 mins and nothing happened. (Normally the program should print a string "Usage : ./myelf password" as i didn't give any arguments). Then i tried with gdbserver on the emulator and a gdb configured for a mipsel processor on the host machine but it didn't work..

I may do something wrong or stupid because i'm quite confused. If anybody can tell me what's wrong in my process or if someone tried to run myelf file, I would know how he did in order to be able to run any program on different machine.

Thank you, have a good day!

Answer 1

Get Ready for an Adventure!

You need a few things for your quest! Let's start at the beginning.

QEMU and GDB

QEMU is an emulator for various architectures. Generally, it's used to emulate an entire PC (i.e. to run a virtual machine). However, for debugging a single program this is not necessary. On Linux, you can use QEMU User-Space emulation.

$ sudo apt-get install qemu qemu-user qemu-user-static

Additionally, the GDB which is installed by default for Ubuntu and similar operating systems does not know anything about other architectures. Luckily, there is a gdb-multiarch packages which does!

$ sudo apt-get install gdb-multiarch

Finally, Linux generally relies on the shebang (#!) at the top of shell scripts to inform it what interpreter to use. For binary files, there is no such standard. In order to fill this void, the binfmt package can be used to look at what type a file is, and automatically invoke the correct interpreter. In our case, it will see that you're trying to run a little-endian MIPS (mipsel) binary and invoke qemu-mipsel.

$ sudo apt-get install 'binfmt*'

Running Statically-Linked Binaries

For a statically-linked MIPSEL binary, this is normally all that would be necessary. However, the one you linked to relies on external libraries. If it were statically linked, you could run it now. You can create an example binary to demonstrate this:

$ echo 'int main() {puts("Hello world!");}' > hello.c
$ mipsel-linux-gnu-gcc -xc -static -o mipsel-test hello.c
$ file mipsel-test
a.out: ELF 32-bit LSB executable, MIPS, MIPS-I version 1 (SYSV), statically linked, for GNU/Linux 2.6.18, BuildID[sha1]=2556cc80429de1ab3116278ac10832d72bd7ebab, not stripped
$ ./mipsel-test
Hello world!

Libraries

Since your chosen binary is dynamically linked, you will need to install libraries like libc.so and ld.so for the appropriate architecture. We also need to tell binfmt where to find them.

Ubuntu provides cross-architecture packages for ARM and AArch64. For example:

$ sudo apt-get install libc6-armhf-armel-cross

Ubuntu (if older than Ubuntu 16.04 / Xenial)

Ubuntu 14.04 and does not provide packages for MIPS. Luckily, Debian (which Ubuntu is based off of) does provide packages, and these packages are compatible with Ubuntu. Un-luckily, Debian does not support little-endian MIPS (mipsel). Lucky us once again, as a different Debian derivative, Embedded Debian (emdebian) does provide those packages.

You can add both repositories to your Ubuntu or other Debian-based distro with the command below. If you were only working with ARM or AArch64, you don't need to do this.

$ sudo apt-get install debian-keyring
$ sudo apt-get install debian-archive-keyring
$ sudo apt-get install emdebian-archive-keyring
$ sudo tee /etc/apt/sources.list.d/emdebian.list << EOF
deb http://mirrors.mit.edu/debian squeeze main
deb http://www.emdebian.org/debian squeeze main
EOF
$ sudo apt-get update

Clean Up

When you are done installing the packages (see below), I highly recommend removing the file emdebian.list which we created earlier. While the Emdebian packages are compatible, apt does weird things and may elect to use a Debian package instead of the one your distro is supposed to. If you need to install more packages later, you can just add it again.

$ sudo rm /etc/apt/sources.list.d/emdebian.list
$ sudo apt-get update

Installing the libraries

Now we can install packages! The package which includes all of the mipsel libraries you need to run the binary you selected is

$ sudo apt-get install libc6-mipsel-cross      # For MIPS-EL
$ sudo apt-get install libc6-armhf-armel-cross # For ARM

If you want to build programs like the sample above, you'll need a cross-compiler.

$ sudo apt-get install gcc-4.4-mipsel-linux-gnu # For MIPS-EL on Ubuntu 14.04
$ sudo apt-get install gcc-mipsel-linux-gnu     # For MIPS-EL on Ubuntu 16.04
$ sudo apt-get install gcc-arm-linux-gnueabihf  # For ARM

Very finally, we need to tell binfmt where the libraries are for mipsel binaries.

$ sudo mkdir /etc/qemu-binfmt
$ sudo ln -s /usr/mipsel-linux-gnu /etc/qemu-binfmt/mipsel # MIPSEL
$ sudo ln -s /usr/arm-linux-gnueabihf /etc/qemu-binfmt/arm # ARM

Now you can run the binary on your system*!

$ ./myelf
Usage: ./crackme password

Debugging

This is the whole point, right?

strace

The quickest thing to do is to be able to run strace on the binary. You can do this with:

$ qemu-mipsel -strace ./myelf
12825 brk(NULL) = 0x00411000
12825 mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x767ca000
...
12825 ptrace(0,0,0,0,0,0) = -1 errno=89 (Function not implemented)
qemu: Unsupported syscall: 4026

You should've seen from running the binary, and from strace, that this binary attempts to call ptrace on itself. This is one of the limitations of qemu-user, is that ptrace is not supported. If the binary needs to ptrace itself, you will need to build a full QEMU system image. I generally use the ones available here. Running QEMU is outside the scope of this answer, but the page I linked to has useful command-lines.

You can easily remove the ptrace call from the binary (it's anti-debugging stuff), and it runs fine. However, I think this is part of the crackme :P.

$./myelf
KO
$ sed -i 's|ptrace|isnanl|' myelf
$ ./myelf
Usage: ./crackme password

QEMU GDB Stub

In order to debug the binary with GDB, you need to launch qemu-mips so that it exposes a GDB stub, and connect from GDB.

$ qemu-mipsel -g 12345 ./a.out &
$ gdb-multiarch ./a.out
(gdb) set arch mips
The target architecture is assumed to be mips
(gdb) set endian little
The target is assumed to be little endian
(gdb) target remote localhost:12345
Remote debugging using localhost:12345
0x00400280 in _ftext ()
(gdb) x/i $pc
  => 0x767cb880    move   $t9, $ra

You can now debug as you normally would. Note that since you're running inside of qemu-user, some commands my not work as expected. In particular, info proc maps doesn't work. You may want to take a look at my pwndbg project, which works around some of these limitations.

Answer 2

You can try to connect remote gdbserver using radare2 tool, with a following line:

r2 -a mips gdb://[address]

Answer 3

qemu-system-mips full system emulation with Buildroot

This method simulate the entire MIPS CPU and can overcome limitations mentioned with userland emulation at: https://reverseengineering.stackexchange.com/a/8917/12321

This script fullys automate everything that is automatable for you: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/1d197f35ee177c6ab3e6a2f518292b8dc4e53431#gdbserver

Buildroot downloads and compiles the entire target filesystem for us, uncluding gdbserver, and also the host QEMU and gdb. Major steps covered below.

The "only" things that are now left now are to:

• cross compile / install all libraries at the correct version for the executable. stdlib is already cross compiled, but you can still have version mismatch problems with it...
• ensure that the dynamic loader is configured correctly. TODO: does Buildroot support customizing it, or can it be patched on the executable itself?

to ensure that you are able to run the executable.

But note that those steps are also required for the userland method.

The major steps done by my script are:

1. Enable gdbserver for on the Buildroot configuration:

   BR2_DEBUG_3=y
   BR2_ENABLE_DEBUG=y
   BR2_OPTIMIZE_0=y
   BR2_PACKAGE_GDB=y
   BR2_PTHREAD_DEBUG=y
   

2. Use BR2_ROOTFS_OVERLAY on the Buildroot configuration to place the executable in the filesystem.

3. Forward a host port to the QEMU target with the command line options (here 45455 to 45455):

   -net user,hostfwd=tcp::45455-:45455
   

4. Launch gdbserver on target:

   gdbserver :45455 myexec
   

5. Launch the cross compiled GDB from host and point it to the cross compiled executable and the forwarded port:

   buildroot/output/host/usr/bin/mips64-linux-gdb \
     -ex 'target remote localhost:45455' \
     -ex 'tb main' \
     -ex 'c' \
     buildroot/output/build/mypackage-1.0.0/myexec
   

Minimal working user mode example

As a quicker and dirtier alternative to full system, you might get away with user mode as described in detail at: https://stackoverflow.com/questions/20590155/how-to-single-step-arm-assembly-in-gdb-on-qemu/51310791#51310791

Answer 4

I want just to add a quick receipt about qemu-user and ptrace.

1. Start qemu-binfmt service on the host to register qemu-user binaries.
2. Use buildah to create amd64 container.
3. Install latest qemu inside this amd64 container, kernel will use them instead of host.
4. Build some toolchain inside amd64 container.
5. Use regular strace to trace your binaries.

For example:

buildah copy 2f64fe2cdbfd /tmp/readdir32.c /tmp/
buildah run --cap-add=CAP_SYS_PTRACE 2f64fe2cdbfd -- sh -c "mips-unknown-linux-gnu-gcc /tmp/readdir32.c -o /tmp/readdir32 && LD_PRELOAD='/usr/mips-unknown-linux-gnu/lib/libc.so.6' strace -v /tmp/readdir32"

write(1, "errno 89 Function not implemente"...

Works perfect.

Full examples are here. Please be aware - test images are not completed, I am still fighting with mips.