17

Are there any tools available to help automate the process of reverse engineering file formats? In particular, I'm interested in tools that use dynamic analysis of an application to parse the format, and less interested in visualization or hex editor-type tools (e.g. 010 Editor).

There is some academic literature on the topic:

Has any of this made it into practice?

9

You don't state the purpose. If you want to fully understand the format and not just automate certain initial aspects of the analysis, it will be very hard to come up with a generic enough tool. This is what domain-specific languages and extensible tools (010 Editor, scripting languages) exist for. I'm not aware of any tools that would "do it all" at the moment.

Tools such as 010 Editor in fact provide additional useful information such as a histogram (giving clues about the entropy, compression/encryption) and give you the binary templates to refine your knowledge about the file format more and more. The most annoying aspect is how limited the scripting is and that you can't, for example, write plugins (imagine being able to decompress bzip2 or deflate streams and such, something commonly found but not at all supported by 010 Editor). One of the major sore points with me and 010 Editor has been that I hit some syntactic limitations of binary templates to express something vital and had to work around that in weird ways (parametrized structs being a major pain), although this is exactly the problem it tries to address with its own DSL. I think we're in dire need of a FLOSS solution (and I was in fact looking into Luaizing frhed at some point).

A file format could be under a layer of encryption or compression and that could be separate per section of the file, not per file. There could be several layers. While I don't want to go as far as to state that it's outright impossible to come up with anything like that, there is a reason why IDA is interactive, for example. In most RCE tasks even the best heuristics can't replace the skills and experience of a seasoned reverse engineer. From experience I would apply the same to reverse engineering of file formats.

I, too, reversed some file formats and usually use a combination of writing a parser in a scripting languages on one hand and writing a binary template for 010 Editor on the other hand. The latter provides a nice fallback, because even if the parsing fails, I can go there, investigate what's going on, adjust and re-run. Lather, rinse, repeat ... you get the idea.

The biggest challenge by far was finding out the more arcane aspects such as integers encoded in 24 bit, sometimes LSB and sometimes MSB first, dates, trees based on xoring two indexes from elsewhere, a few bits being used from one byte and a few from another having particular meaning... that stuff. And I really cannot imagine how you would fill those gaps without reverse engineering the code reading/writing the format originally. Lacking any documentation/standard, it's the best reference you got.

Also, don't underestimate the power of visualization. Of course a single angle, such as a histogram, only provides little information. But there are plenty of interesting algorithms to apply to unknown data and see structures.

  • 2
    I should clarify: I'm not looking for something that automates everything -- as you say, a tool like IDA would be best. But there's a whole lot of room for improvement between fully manual and fully automated. Even a tool that just annotated a file with the code that parsed each range of bytes would be hugely valuable. – Brendan Dolan-Gavitt Mar 27 '13 at 19:09
  • @BrendanDolan-Gavitt: I totally agree. Alas, I'm not aware of any. It would be dependent on the architecture and moreover the platform to be meaningful. I could, however, imagine that something like that exists to parse the output of strace, for example. – 0xC0000022L Mar 27 '13 at 19:16
  • Also want to add that I definitely don't discount visualization; I've done a little bit of work toward that here and here. Those are both portions of the PDB file format, and it is very helpful to visually see the access patterns to get an idea of where structure boundaries are. – Brendan Dolan-Gavitt Mar 27 '13 at 19:43
2

The academy is very far from industry, which is always many-many years ahead. I think there is nothing 'magical' to reverse engineer file formats. Indeed, I reversed various file formats and the very 1st time I did this I tried to find if there was such a tool. But, there was nothing at that time and I don't think there is anything at all today.

It's possible to write a tool that 'automagically' detects some patterns and creates an initial structure to start working on without the need to start from zero but I don't think anything else exists.

  • 1
    It's a shame if no tools exist -- I can think of lots of automation that would be quite useful, particularly if we bring in some data flow analysis! Perhaps another thing to add to my todo list... – Brendan Dolan-Gavitt Mar 27 '13 at 17:57
2

Not a direct answer to your question, but I guess it could help.

There have been more attempts at analyzing unknown network protocols and I guess some ideas and techniques could be applied to analyzing unknown file formats. Some of the papers are:

There is also Netzob tool:

Netzob is an open source tool for reverse engineering, traffic generation and fuzzing of communication protocols. It allows to infer the message format and the state machine of a protocol through passive and active processes. The model can afterward be used to simulate realistic and controllable trafic.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.