16

A key tool in reverse engineering is a good disassembler, so to ensure that a disassembler is performing properly, are there any good test suites available for use to test the correctness of a disassembler? Are these architecture specific, or can they be configured to work across multiple object architectures? A good test should include checking the more obscure architecture instructions and malformed portable execution files.

Here is one specifically for i86 that I have seen. Are there any that are modular across architectures?

Improve this question
2
  • 1
    When you say "correctness of a disassembler", do you mean correctness of an instruction decoder, or, do you mean correct control flow graph recovery and other kinds of high-level semantic information?
    – Andrew
    Mar 20, 2013 at 16:17
  • Well, which is the more interesting question? I'm guessing the later, correct flow graph recovery...
    – WilliamKF
    Mar 20, 2013 at 16:25

5 Answers 5

Reset to default
14

There is a paper called "N-version Disassembly: Differential Testing of x86 Disassemblers" (PDF) by Roberto Paleari, Lorenzo Martignoni, Giampaolo Fresi Roglia, and Danilo Bruschi which compares several x86 disassemblers in a formal fashion.

To quote the papers abstract:

The output of a disassembler is used for many different purposes (e.g., debugging and reverse engineering). Therefore, disassemblers represent the first link of a long chain of stages on which any high-level analysis of machine code depends upon. In this paper we demonstrate that many disassemblers fail to decode certain instructions and thus that the first link of the chain is very weak. We present a methodology, called N-version disassembly, to verify the correctness of disassemblers, based on differential analysis

Not sure if this is slightly off topic to your question but may be of interest to you.

Improve this answer
1
8

In a lot of papers I've read, decompilation tool authors use the SPEC benchmarks to measure the effectiveness of their decompiler. This produces kind of a holistic view of how well the system works, from instruction decoding to control flow recovery. Those benchmarks aren't free or open source though.

The GCC and clang compilers also ship with benchmarks for testing. Those might be worth investigating.

Also, Regehr's csmith project can generate arbitrary C programs for compiler testing/fuzzing. This could be useful for testing decompilers and binary analysis systems?

Improve this answer
1
  • 1
    FWIW we do use cmith for testing the Hex-Rays decompiler, and it does help.
    – Igor Skochinsky ♦
    Mar 20, 2013 at 17:04
6

The gas testcase suite that you link to is not only for i386. The parent directory contains test cases for x86-64, arm, alpha, and many other architectures.

Improve this answer
5

I made a file containing all x86 opcodes (at the time) and many encoding forms, that helped me to actively find many bugs in many disassemblers (IDA, Hiew, BeaEngine, OllyDbg, XED, hte...), particularly lesser known forms and undocumented ones.

(slightly off-topic maybe - not a ready-made test suite)

Improve this answer
2
  • Did you file bugs for all of them? Did they get fixed?
    – Janus Troelsen
    May 1, 2013 at 20:01
  • 2
    I did, they were.
    – Ange
    May 1, 2013 at 20:16
2

The radare2 project uses an extensive test-suite for each of its disassembler engine, along with more specific tests, like formats, its own analysis capabilities, …

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.