How can the problem posed for disassembly by the mixture of executable code and literal pools in ARM firmware be overcome?

Question

Context and Background Information

Part 1: ARM firmware compared to firmware for other architectures

I looked at the ARM firmware in these two questions:

• Running a binary identified as an ARM excutable by binwalk --disasm
• Approach to extract useful information from binary file

and was baffled by the mixture of code with data, since I have looked at firmware for other architectures such as Ubicom, MIPSEL and MIPS and did not observe the same kind of intermingling of code and data as in the ARM firmware.

Here are some visualizations of various binaries for context:

MIPSEL eCos firmware from Analysing eCos image

<------------------------ code -------------------------><---- images, ASCII---->

Unknown instruction set, uC/OS firmware from lzma: File format not recognized [Details enclosed]

<-------------------------- code??? ------------------------><-- ASCII -->

Ubicom32 SlingBox firmware from Need help extracting YAFFS from firmware .bin file

<--------------------------- code -----------------------> <- data ->< code >

A pattern appears to emerge, no? The executable code in the firmware images presented here is not so difficult to differentiate from data because it largely resides in 1 or 2 large, contiguous blocks.

However, things are not so straightforward with the ARM firmware:

32-bit ARM firmware from Running a binary identified as an ARM excutable by binwalk --disasm

Where is the code? Where is the data? There is no easily discernible separation like in the other firmware binaries.

Thumb-2 (16-bit + 32-bit) ARM firmware from

The situation is similar here.

When the visualizations are juxtaposed, the differences become very obvious:

The non-ARM firmware binaries in this sample have fairly clear code and data separation, which is helpful if we want to disassemble the code. However, identification of code and data looks to be quite problematic in the case of the ARM firmware binaries.

Part 2: ARM Literal Pools

So why do the ARM firmware binaries look so different from the others? Why is there no clearly discernible separation of code and data? The answer seems to involve ARM literal pools.

I'm including information about ARM literal pools here because it seems relevant and because it took forever for me to find out about them on my own, so hopefully this saves others some time.

According to Ian Cook (emphasis mine),

...mixing of code and data is actually very common with ARM code.

ARM instructions are fixed sizes (4 bytes on ARM, 2 bytes for THUMB) and have insufficient space to encode an 32 bit immediate value. Instead, ARM compilers usually do two things -

• they place 32 bit immediate constants straight after the instructions for the function in which the constants are needed, and

• they use PC-relative load instructions in the function to put these constants into registers.

This is similar to the information I found in the article The trouble with literal pools:

So, what is a literal pool? I’m glad you asked. The literal pool is an area of memory (in the text segment), which is used to store constants. These constants could be plain numerical constants, but their main use is to store the address of variables in the system. These addresses are needed because the ARM instruction does not have any instructions for directly loading (or storing) an address in memory. Instead ldr and str can only store at a ±12-bit offset from a register. Now there are lots of ways you could generate code with this restriction, for example, you could ensure your data section is less than 8KiB in size, and reserve a register to be used as a base for all data lookups. But such approach only works if you have a limited data section size. The standard approach that is taken is that when a variable is used its address is written out into a literal pool. The compiler then generates two instructions, first to read the address from this literal pool, and the second is the instruction to access the variable.

So, how exactly does this literal pool work? Well, so that a special register is not needed to point at the literal pool, the compiler uses the program counter (PC) register as the base register. The generated codes looks something like: ldr r3, [pc, #28]. That codes loads a value at a 28-byte offset from the current value of PC into the register r3. r3 then contains the address of the variable we want to access, and can be used like: ldr r1, [r3, #0], which loads the value of the variable (rather than the address) into r1. Now, as the PC is used as the base for the literal pool access, it should be clear that the literal pool is stored close enough to the code that needs to use it.

To ensure that the literal pool is close enough to the code using it, the compiler stores a literal pool at the end of each function. This approach works pretty well (unless you have a 4KiB+ function, which would be silly anyway), but can be a bit of a waste.

However, in the ARM documentation for literal pools, it is stated that

The assembler uses literal pools to hold certain constant values that are to be loaded into registers. The assembler places a literal pool at the end of each section. The end of a section is defined either by the END directive at the end of the assembly or by the AREA directive at the start of the following section. The END directive at the end of an included file does not signal the end of a section.

I don't know jack about ARM, so this is confusing. I might make this a separate question. Anyway, the point is that literal pools appear to be the reason why code and data are mixed together in the ARM firmware visualized earlier. There is also something called scatterloading, but I don't know if this has anything to do with the problem described here.

Question

How can the problem posed for disassembly by the admixture of executable code and literal pools in ARM firmware be overcome?

According to Ian Cook (from the same answer as before) (emphasis mine),

It's usual to see the immediate constants stored after the function in order of the addresses of the functions that access them. So, in this case, the function ends with the instruction at 0x00000F5C and the immediate constants would appear to span the address range 0x00000F60 to 0x00000FFF and I would usually expect the following function to begin at address 0x00001000.

Knowing this it would be possible for a disassembler to identify this pattern and automatically skip the associated data.

Yes, how? I tried disassembling both the ARM 32-bit Panasonic picture frame firmware and the ARM Thumb-2 SMOK-X Cube contraption firmware with radare2 but it seemed like data was being disassembled along with code.

Since the location of literal pools after functions seems to be a common feature of ARM binaries, are existing disassemblers able to differentiate between code in functions and the data in their adjoining literal pools even when there aren't any symbols in the binary? This isn't the case with Capstone as far as I know.

Answer 1

This is indeed a problem for linear sweep disassemblers; recursive descent ones are mostly unaffected since they won't try to disassemble data if there are no code flows into it. However, there may be some heuristics/cheap tricks that might help even linear sweep disassemblers. Some that come to mind:

• while disassembling instructions, look for LDRs with PC-relative operands (as well as ADR instructions) and remember the target addresses; stop disassembly when reaching those addresses.

• in ARM mode, stop disassembly when decoded instructions have nonsensical conditions, especially when there is no preceding flag-setting instruction.

• gather some stats (e.g. n-gram frequencies) from binaries which do have symbols and use them to tell code from data.

  In the end, I don't think there is a good generic solution for this problem, so it may be even easier to just tune the approach for each specific target.