Can an OpenWrt FIT kernel be virtualized in qemu?

Question

I have a router I'd like to virtualize in order to do some dynamic analysis. I could virtualize individual binaries with qemu, but I'd like to have the whole ecosystem up and running.

I'll start here:

$ file firmware.bin
firmware.bin: data

Okay, binwalk to the rescue.

$ binwalk -e firmware.bin
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
4755          0x1293          UBI erase count header, version: 1, EC: 0x0, VID header offset: 0x800, data offset: 0x1000

Great, I've got something. The UBI is a little funky, but with a little legwork, I've got two files.

$ file rootfs.ubfs
rootfs.ubfs: Squashfs filesystem, little endian, version 1024.0, compressed, 7852451465615376384 bytes, 906821632 inodes, blocksize: 1024 bytes, created: Wed Mar 26 08:01:38 2092

$ xxd kernel.ubfs | head
00000000: d00d feed 0043 8904 0000 0038 0043 8598  .....C.....8.C..
00000010: 0000 0028 0000 0011 0000 0010 0000 0000  ...(............
00000020: 0000 006c 0043 8560 0000 0000 0000 0000  ...l.C.`........
00000030: 0000 0000 0000 0000 0000 0001 0000 0000  ................
00000040: 0000 0003 0000 0004 0000 005c 6298 dbf6  ...........\b...
00000050: 0000 0003 0000 0027 0000 0000 4152 4d20  .......'....ARM
00000060: 4f70 656e 5772 7420 4649 5420 2846 6c61  OpenWrt FIT (Fla
00000070: 7474 656e 6564 2049 6d61 6765 2054 7265  ttened Image Tre
00000080: 6529 0000 0000 0003 0000 0004 0000 000c  e)..............
00000090: 0000 0001 0000 0001 696d 6167 6573 0000  ........images..

Looks like my version of file isn't picking up on the "kernel.bin" file, but we can see that it's an OpenWrt FIT.

I'm not quite sure where to go from here. kernel.bin seems to maybe be a dtb or "Device Tree Blob". I think I need all of the information stored in the dtb in order to properly virtualize. My understanding is that the dtb lays out physical memory addresses and interrupt request lines needed to communicate with other devices, some of which may be virtualizable by qemu. A dtb file can even be passed to qemu with the -dtb option.

At this point, I'm still thinking I need a kernel. I can build one from the GPL source of the same model, but different hardware revision. Like usual, GPL sources aren't out for the latest firmwares / models.

So where is the kernel? A binwalk on kernel.bin and a strings later, I find a recognizable kernel string in the kernel.bin file.

$ strings E4 | grep "too many orphaned sockets"
6TCP: too many orphaned sockets

Makes sense. The 0xe4 bytes extracted are probably a decompression stub. Then I was able to grep on a decompressed kernel image.

This isn't a kernel image. There's an openwrt FIT stub in front of the decompression stub? That makes sense.

So there's a rootfs, and a dtb. The kernel appears to be in embedded in the dtb. The logical thing would be to figure out the OpenWrt FIT format. My google-fu isn't helping much here.

Can an "OpenWrt FIT" kernel be virtualized in qemu? I'm having trouble finding documentation on the format.

Answer 1

I don't know anything about OpenWrt FIT format, but I've built an OpenWRT VM to run on OpenStack in the past, so virtualising your router image might be doable.

My first thought was that you build an OpenWRT VM image then copy the file content from the router image that you have into the new QCOW.

Here's my notes on the build, it was a couple of years ago so ymmv:

## http://hackstack.org/x/blog/2014/08/17/openwrt-images-for-openstack/
## https://centos.pkgs.org/7/centos-x86_64/zlib-static-1.2.7-17.el7.x86_64.rpm.html

git clone git://git.openwrt.org/openwrt.git
cd openwrt
echo "src-git dtroyer https://github.com/dtroyer/openwrt-packages" >>feeds.conf.default
./scripts/feeds update -a
./scripts/feeds install -a

sudo yum -y groupinstall development
sudo yum -y install ncurses-devel zlib-devel openssl-devel
yum -y install http://mirror.centos.org/centos/7/os/x86_64/Packages/zlib-static-1.2.7-17.el7.x86_64.rpm

make defconfig

make menuconfig
  Target System: x86
  Subtarget: KVM guest
  Target Images
    [*] ext4
    (128) Root filesystem partition size (in MB)
    (/dev/vda2) Root partition on target device
  Base System
    {*} block-mount
    <*> rc.cloud

make -j <cpu_count>

# troubleshoot
make V=sw

gzip -dc bin/x86/openwrt-x86-kvm_guest-combined-ext4.img.gz >openwrt-x86-kvm_guest-combined-ext4.img
sudo kpartx -av openwrt-x86-kvm_guest-combined-ext4.img
mkdir -p imgroot
sudo mount -o loop /dev/mapper/loop0p2 imgroot
sudo chroot imgroot /bin/ash
# in chroot:
  # set root password
  sed -e '/^root/ s|^root.*$|root:\!:16270:0:99999:7:::|' -i /etc/shadow
  # configure lan interface with dhcp - doesn't seem to work?
  uci set network.lan.proto=dhcp; uci commit
  # Configure opkg:
  sed -e "s|http.*/x86/|http://bogus.hackstack.org/openwrt/x86/|" -i /etc/opkg.conf
sudo umount imgroot
sudo kpartx -dv openwrt-x86-kvm_guest-combined-ext4.img