15

A key tool in reverse engineering is a good disassembler, so to ensure that a disassembler is performing properly, are there any good test suites available for use to test the correctness of a disassembler? Are these architecture specific, or can they be configured to work across multiple object architectures? A good test should include checking the more obscure architecture instructions and malformed portable execution files.

Here is one specifically for i86 that I have seen. Are there any that are modular across architectures?

  • 1
    When you say "correctness of a disassembler", do you mean correctness of an instruction decoder, or, do you mean correct control flow graph recovery and other kinds of high-level semantic information? – Andrew Mar 20 '13 at 16:17
  • Well, which is the more interesting question? I'm guessing the later, correct flow graph recovery... – WilliamKF Mar 20 '13 at 16:25
13

There is a paper called "N-version Disassembly: Differential Testing of x86 Disassemblers" (PDF) by Roberto Paleari, Lorenzo Martignoni, Giampaolo Fresi Roglia, and Danilo Bruschi which compares several x86 disassemblers in a formal fashion.

To quote the papers abstract:

The output of a disassembler is used for many different purposes (e.g., debugging and reverse engineering). Therefore, disassemblers represent the first link of a long chain of stages on which any high-level analysis of machine code depends upon. In this paper we demonstrate that many disassemblers fail to decode certain instructions and thus that the first link of the chain is very weak. We present a methodology, called N-version disassembly, to verify the correctness of disassemblers, based on differential analysis

Not sure if this is slightly off topic to your question but may be of interest to you.

8

In a lot of papers I've read, decompilation tool authors use the SPEC benchmarks to measure the effectiveness of their decompiler. This produces kind of a holistic view of how well the system works, from instruction decoding to control flow recovery. Those benchmarks aren't free or open source though.

The GCC and clang compilers also ship with benchmarks for testing. Those might be worth investigating.

Also, Regehr's csmith project can generate arbitrary C programs for compiler testing/fuzzing. This could be useful for testing decompilers and binary analysis systems?

  • 1
    FWIW we do use cmith for testing the Hex-Rays decompiler, and it does help. – Igor Skochinsky Mar 20 '13 at 17:04
6

The gas testcase suite that you link to is not only for i386. The parent directory contains test cases for x86-64, arm, alpha, and many other architectures.

5

I made a file containing all x86 opcodes (at the time) and many encoding forms, that helped me to actively find many bugs in many disassemblers (IDA, Hiew, BeaEngine, OllyDbg, XED, hte...), particularly lesser known forms and undocumented ones.

(slightly off-topic maybe - not a ready-made test suite)

  • Did you file bugs for all of them? Did they get fixed? – Janus Troelsen May 1 '13 at 20:01
  • 2
    I did, they were. – Ange May 1 '13 at 20:16
2

The radare2 project uses an extensive test-suite for each of its disassembler engine, along with more specific tests, like formats, its own analysis capabilities, …

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.