What tools exist for excavating data structures from flat binary files?

Question

Problem Statement

I have a file composed entirely of data structures; I've been trying to find a tool that will enable me to open this file, and declare (perhaps) a type and offset such that i may work with the presumed primitive data type individually.

e.g. I declare the 4 bytes located at offset 0x04 to be a 32-bit unsigned integer, and would like to inspect the value at this location (read as big-endian perhaps) and then work with this integer individually (perhaps see what it looks like encoded as a 4-byte ascii string and attempt to read it, etc.)

Specifics

I have a 4096 byte file containing C-structs with member elements as integers ranging from 16-64 bits in length; the following is an example:

struct my_struct {
uint_32 magic
} // sizeof(my_struct) == 0x04

In this case, magic = 'ball', and so when the file is opened in a text editor it reads as 'llab...', and obviously can also be represented as a 32-bit integer

Question

Is there a tool that enables static analysis of flat data structure files?

What I've considered thus far as a solution

I've considered writing a command line tool in Python to do this, but if something already exists I'd prefer to save time, and perhaps learn more about this topic by using a tool designed by someone more experienced. If it seems to you that I am going about this incorrectly (this is my first serious exploration into this kind of reversing) please guide my understanding, thanks.

Where I have already researched

Googled 'reverse engineering tools' and browsed the links

Checked wikipedia's reverse engineering pages

Tried some first principles reasoning

Checked pypi

Results

There are three completely valid and correct answers, but I've marked the most detailed and least expensive of them as correct, because it is the most accessible to members of the community reviewing this question.

Answer 1

010 Editor's Binary Templates feature is ideal for this type of work. It not only supports static struct definitions, but also allows for dynamically sized struct fields and supports a powerful scripting language.

Answer 2

I use Hex Workshop for this. It has a bunch of useful flat-file reversing features, but my favourite is that it lets you declare structures in C-style syntax and load them on top of a file. It's not free, but it's more than worth the $90 price tag.

Features I find most useful:

• C-style struct syntax, supporting various arrays, string types, bitstrings, validity checks, etc.
• Customisable data inspector (useful if you only want to see data as a few types)
• Diffing
• Offset display (shows address offsets, selection size, etc.)
• Sequence highlighting (like "highlight all" on find)
• Bitwise operations, checksums, etc.

Answer 3

IDA can be used for working with data-only files. You can convert bytes to data items (bytes/words/dwords/qwords/floats/strings etc.), group them into structures or arrays (or arrays of structures), represent integers as offsets, add names and comments and so on.

Here's an example of some random BMP file represented in IDA:

0000 BmHeader     db 'BM'     ; Signature
0002              dd 146h     ; Size
0006              dw 0
0008              dw 0
000A              dd offset pixel_array ; offset to image data
000E ; DIB header
000E              dd 40       ; size of this header
0012              dd 33       ; bitmap width
0016              dd 33       ; bitmap height
001A              dw 1        ; number of color planes
001C              dw 1        ; bits per pixel
001E              dd 0        ; compression: none
0022              dd 108h     ; size of image data
0026              dd 4724     ; horizontal resolution (pixels per meter)
002A              dd 4724     ; vertical resolution (pixels per meter)
002E              dd 0        ; colors in the palette
0032              dd 0        ; number of important colors
0036 ; Color table
0036              dd 0, 0FFFFFFh
003E ; pixel data
003E pixel_array  db 1, 73h, 13h, 6, 80h, 3 dup(0), 7Dh, 4Bh, 0E0h, 56h

Answer 4

Hexinator has a similar feature to binary templates of 010 Editor that is called a "grammar". It allows to insert numbers, strings, structs, and binary blobs. If that's not enough, it has scripting capabilities in Python and Lua.

The values can then be edited nicely (e.g. in decimal instead of hex). The hex area can be highlighted.

Drawbacks:

At the time of writing it seems to have issues with more than one open grammar + one open file. When opening a second file for the same grammar, it crashed. Save early and save often.

Screenshot of a partially analyzed file:

Answer 5

I use my Binary Editor for this kind of task http://www.nyangau.org/be/be.htm