48

Let's say I found 'some' file (might be an executable, might be data, or something else) and want to run or read it. I open this file in a text editor, but the format isn't readable. Examples include: Java class, Windows executable, SQLite database, DLL, ...

I do know the file format, if we can trust the extension.

Is there somewhere a site or database with a lot of information about a lot encrypted or binary file formats? Information should include:

  • File use
  • File layout and structure
  • Eventually programs that can read or execute the file

So I'm not looking for a way to identify the format of the file. I already know the file format, but need to have information about that format. When is the format used (in what applications), what's the format's structure?

  • 4
    I would guess that all these data are dispatched in the wild. I don't see any group capable to maintain a coherent collection of all the specification of all formats. But, I might just be ignorant as well. :) – perror Apr 4 '13 at 7:59
  • 2
    So you have a file with an undocumented format? I think the question is still worded a bit ambiguous even now. Also, the closes you'd get is Wotsit and similar sites. Everything else would more likely be a site covering domain knowledge about whatever the specific format is used for. – 0xC0000022L Apr 4 '13 at 17:23
  • 1
    @0xC0000022l yeah, I intended to keep the question as broad as possible, but that didn't go well. I don't have a particular example file I'm looking at at the moment, I just wanted to know if there was a central place with file format information. Wotsit is a very good one indeed, as mentioned in this answer. – Keelan Apr 4 '13 at 17:25
  • I find the question interesting, however, all good sites and libs I can think of for now have been mentioned already, so are you actually now (because of the bounty) looking for a precise methodology? – Ange Apr 21 '13 at 15:41
  • @Ange Hachoir is the best I've seen up to now, however, since many things have been mentioned, perhaps a CW list would be better. If not, I'll just use the bounty to reward an existing answer. Especially the Hachoir thing seems suitable for that IMO. – Keelan Apr 21 '13 at 15:43

12 Answers 12

36
+50

I am a developer for the Hachoir project, which aims to describe the format of any file down to the bit-level. Currently it has parsers for a number of well-known formats, including Java classes, Windows executables, MP3 files, etc. The full list of parsers can be found in the subdirectories here.

When working on a new parser for hachoir_parser, it is often necessary to find information about a file format. There's no single source describing every format (even as Hachoir aspires to be this source, it is not nearly comprehensive enough). Generally, the first step is just to search for <file format name> file format, e.g. java class file format, and look for documentation on official sites (for java classes, this turns up Oracle's documentation, which should be all you need). If there are no official sites, you may still turn up some documentation from someone who has worked on the format in the past.

For common file formats, this turns up the format specification you want about 90-95% of the time. Larger software companies, like Oracle and Microsoft, post their file format specifications online for interoperability purposes. For example, you can find documentation for PE (Windows EXE/DLL), MS Office formats (XLS, PPT, DOC), and other Microsoft formats by browsing or searching MSDN.

For multimedia formats, the Multimedia Wiki is a great resource. They also cover some game file formats as well.

For a less common file format, for which I do not find a specification (or suitable description) from Googling, my approach is usually to find an open-source program that does understand the file, and either locate their format specification source (if described in a commit or README), or read their source code directly to understand the file format.

If there are no open-source programs for the file, and no openly available descriptions of the file format online, the file format is probably quite obscure. For game files (in which many developers insist on using their own proprietary formats), I've found XeNTaX to offer some good pointers and a good community to help figure out the formats. With other kinds of formats, you may have to start examining the samples you have to compare the byte fields and elucidate their function. If you have a program that accepts these files, you can try changing the fields methodically to determine what effects they have on the program's output. This is ultimately the "real" reverse-engineering work, and I think it is not within the scope of this answer.

  • 2
    I'm a big fan of the Hachoir project. I have been using it for years. Keep up the great work. – alexanderh Apr 8 '13 at 20:43
  • 1
    One of the guys on XentaX also wrote a script/framework called QuickBMS which is quite amazing. It sounds like it may be useful for Hachoir. aluigi.altervista.org/quickbms.htm – Adam Baxter Aug 19 '13 at 6:34
27

The file(1), and the underlying libmagic(3), command fingerprints files based on the file content. For example:

$ file test.c
test.c: ASCII text
$ file test.exe
test.exe: PE32 executable for MS Windows (console) Intel 80386 32-bit
$ file test
test: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, not stripped
$ file database.sqlite
database.sqlite: SQLite 3.x database

This information can then be used to search the Internet for the actual standard, RFC, ...


Another resource could be the 010 Editor binary templates, which include a lot of different file formats and which you can customize or develop yourself from scratch.


There is a German book I know called Dateiformate (German for: file formats).


Wikipedia - List of file formats contains a lot of links with information about certain file formats

  • 3
    If this was the answer wouldn't the question be a duplicate of this? I think he wants a database of specifications of file formats. Not just magic numbers. – Peter Andersson Apr 4 '13 at 15:38
  • 3
    Exactly. Your answer is useful, but not an answer to the question. I do know the file format, I need to have information about that format - how the file is structured, for example. – Keelan Apr 4 '13 at 15:57
  • You are right. Kind of misinterpreted the question. I just remembered that I saw a book with a lot of file formats in the "library" of my local hackerspace. It's a german book but maybe somebody can find the english version. I Updated my answer. – samuirai Apr 8 '13 at 16:25
11

I like file to determine the type of file from the header magic and Wotsit for standard file formats/documentation/reversed file format structures by other people.

Fileformat.info was suggested as another resource for those that don't like Wotsit or feel it's dated.

  • 1
    Wotsit is not really a good choice: downloading problems (in effect now!), poorly organized collection. – user539484 Apr 5 '13 at 10:54
  • 1
    @user539484: if you don't like the organization, use the site: or inurl: operators of a certain search engine with G to have an advanced search tool. – 0xC0000022L Apr 5 '13 at 17:05
  • 1
    @0xC0000022L, like it of some help with wotsit.org... It has been stuck in 1990s. fileformat.info collection is almost as broad and way friendlier for the reader. To demonstrate a frustration can be caused by wotsit i'd suggest an attempt to obtain New Executable related publications from wotsit collection. – user539484 Apr 5 '13 at 18:19
  • 1
    @user539484 - Then why not just add it in? The other answers doesn't give you any resource at all. Just "here's a tool that sort of works". – RobotHumans Apr 5 '13 at 19:01
  • 2
    @Call me V, fileformat.info was listed already. I also use Wikipedia, sometimes their file layout articles are surprisingly good and technical. – user539484 Apr 5 '13 at 21:16
10

The Lets Solve the File Format Problem wiki covers quite a good number of obscure formats and is worth checking out if Wikipedia isn't doing the job for you. It's also a good place to contribute very detailed or obscure stuff that Wikipedia might deem out of scope.

You may find help or advice from the library and archival digital preservation community which has been taking on problems of this nature for a number of years. The Open Planets Foundation is a good place to start. In particular, the community (who I work with) collects examples of difficult or damaged file formats on a Github based Format Corpus. Contributing to this can be useful in attracting advice from experts, and of course this helps to develop a useful resource for developers of digital preservation tools (for testing and validation). Files must be CC0 however!

Finally, the other avenue to explore further would be the digital forensics community. The National Software Reference Library, for example, has some useful resources that could help around the fringes of your challenge.

6

If I understand correct, You need to identify the file? First of all - extension is not 100% guarantee that file is what it pretend to be. You easily can rename binary.dll to binary.dat or binary.hhh and still create file association or load it as DLL.

So - to identify file, You may use TrID of Marco Pontello.

Also - hex editor is a handy way to look into streams and all kind of dumps.

When You identify the file - Google will help You to find correct way to edit\open\execute the file.

  • 1
    Thank you for telling about TrID, I didnt know about such a great program :) – Jake Badlands Apr 4 '13 at 8:46
  • 1
    Always welcome, Jake :) – Denis Laskov Apr 4 '13 at 8:48
  • 2
    Thanks for your reply, but I don't have to identify the file. I already know the format (let's assume the extension is right, for now) but need to have information about that format - for example, how the file is structured. – Keelan Apr 4 '13 at 15:58
  • 1
    Then, as I said, Google will help You to find proper RFC or details per file format. as for .DLL - it structured almost similar to .EXE, except for export part of PE header. – Denis Laskov Apr 4 '13 at 16:08
5

Like in many things, if you need more information, Wikipedia is the answer. It only has detailed information for some formats, but usually has links to more in-depth technical descriptions of the formats. You kind of have to guess what article the file format would be under, but in many cases you can find good information on (at least) many of the more common formats. It's not ideal - perhaps it'd be a useful project to make a new "file format" wiki - but it does have a decent amount of information.

It's far from an ideal source for this kind of data, but in terms of one-site resources it's one of the better options, and is certainly a good starting point.

A few examples:

5

Check out the Hachoir project. It is exactly what you are looking for. Hachoir-subfile is great because you can pass it a file stream and it will identify all embedded file types, size (if can be calculated) and their offset. Hachoir and dd are two of my favorite tools for carving files out of data streams. If you are running ubuntu type sudo apt-get install hachoir, then run hachoir-subfile <file_stream>.

If you need to dig a little deeper into the stream you can use hachoir-urwid. It will break the file up into known structures.

  • 1
    I don't see where Hachoir gives information about a specific format? It's just a file identifier. – Keelan Apr 8 '13 at 14:46
  • 4
    [Disclaimer: I'm a Hachoir developer.] Hachoir parses files in several known formats to present a view of the file's contents, along with descriptions to identify each field in the file. Hachoir's goal is to understand a file down to the bit level. So, Hachoir is used for much more than file identification -- it can do field extraction, field modification, and will provide textual descriptions of every field in the file. – nneonneo Apr 8 '13 at 15:44
4

I am afraid that there is no website on this topic yet :(
From my experience, the situation when you have the file but do not know its type
is not often, but when it happens it can be annoying and time-consuming to solve.
Basically, you need to use hex editor to spot characters which could give you a clue,
and download example files of suspected file types to analyze them and find similarities.
This approach is working for me in most cases.

  • 1
    Sorry, I wasn't clear. I do know the file type (based on the extension), but need information on the file format in order to be able to read or execute it. – Keelan Apr 4 '13 at 8:04
  • @CamilStaps So if I understood correctly, you are talking about such kind of situation when you have a file (e.g. DLL) but do not know what is its purpose (e.g. which program is using it) ? – Jake Badlands Apr 4 '13 at 8:09
  • 1
    Yes, that, and what the purpose of that specific file is. So for example, I would like to see what that DLL file does. Therefore, I need information about the file format, and this doesn't provide enough information. – Keelan Apr 4 '13 at 8:21
  • 1
    To know, what DLL file does, You need to disassemble it, like any other binary. :) – Denis Laskov Apr 4 '13 at 8:35
  • 2
    I only mentioned DLL as an example. I would like to know if there is a standard place one goes to when he needs to have detailed information about a file format. – Keelan Apr 4 '13 at 15:59
3

Most file formats have headers, such as P K 0x03 0x04 for zip files or 0x89 P N G 0x0D 0x0A 0x1A 0x0A for PNG files.

So, most of time, you just need to read the first bytes to know the format of the file.

You'll find plenty of file format descriptions on http://www.fileformat.info/format/all.htm ­­­

3

About multimedia formats: http://wiki.multimedia.cx/index.php?title=Main_Page

Signsrch will find patterns in a file: http://aluigi.altervista.org/mytoolz.htm, like binwalk: http://code.google.com/p/binwalk/

1

DROID from the National Archives of the Uk is also useful for identifying file formats.

0

This is the tool I have been using when I have needed to recognize a file format or files inside a (big) dump. It has a big signature BD file that you/people can contribute to.

Signsrch 0.2.4 (signsrch)

tool for searching signatures inside files, extremely useful in reversing engineering for figuring or having an initial idea of what encryption/compression algorithm is used for a proprietary protocol or file. it can recognize tons of compression, multimedia and encryption algorithms and many other things like known strings and anti-debugging code which can be also manually added since it's all based on a text signature file read at runtime and easy to modify. supports multithreading, scanning of folders using wildcards, scanning of processes, conversion of the executables offsets in memory offsets, loading of custom signature files and their automatic checking for avoiding errors, automatic finding of the instructions that reference the found signatures (like "Find references" of Ollydbg) and the launching of an executable placing an INT3 byte at the desired memory offset (for example one of those retrieved with the -F option, watch the Video setion for an example). the tool supports 8, 16, 32 and 64 bits, float and double plus automatic CRC table creation and C style strings.

Source: http://aluigi.altervista.org/mytoolz.htm

  • As the OP commented on five similar answers: no need for indentifying, he already knows what the file type is. – usr2564301 Aug 27 '16 at 1:23
  • The point is to help other that might have the same question later on. – Alexandre Dumont Aug 28 '16 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.