I am a developer for the Hachoir project, which aims to describe the format of any file down to the bit-level. Currently it has parsers for a number of well-known formats, including Java classes, Windows executables, MP3 files, etc. The full list of parsers can be found in the subdirectories here.
When working on a new parser for hachoir_parser, it is often ...
The file(1), and the underlying libmagic(3), command fingerprints files based on the file content. For example:
$ file test.c
test.c: ASCII text
$ file test.exe
test.exe: PE32 executable for MS Windows (console) Intel 80386 32-bit
$ file test
test: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, ...
[Complete ReEdit3] further progress & shortening the text to fit the 30KB limit
First some input how I got here (for future readers trying to do the same for different format).
Image Data size
Comparing provided background image to its raw image size xs*ys reveal direct dependence Which implies no compression or one that has always the same pixel to ...
I've been dissecting the firmware for another type of embedded device for a while and thought I'd see if I could find anything out. After a few hours I figured it out! There is a hard way and an easy way that I found only after digging the hard way. This is a long post, but I hope it will help others in similar ventures.
A little Googling and I found ...
I downloaded abc chant.notebook from the site cb88 linked to:
$ file "abc chant.notebook"
abc chant.notebook: Zip archive data, at least v2.0 to extract
$ unzip -t "abc chant.notebook"
Archive: abc chant.notebook
testing: images/temp(1).png OK
... about 200 similar lines ...
testing: attachments/Zachary.JPG OK
No errors detected in ...
Well, obviously the particulars will very much depend on the particulars of the file format and what you expect to achieve in general. However, some steps will largely be the same. One thing you could do is:
try hard to find all kinds of clues about the format. This can be a small note in some bulletin board or the cached copy of some year old website that ...
Looking at it in OllyDbg it looks like a heavy task. Looks like a custom database with encrypted and (custom?) compressed data. This or the like would usually be the case in such applications. A flat file with structured data is not part of this one.
Anyhow. As a starter:
A quick check after trying out some general compression tools like 7z or binwalk, (...
You can try Netzob tool. This is a tool dedicated to reverse engineering protocols.
You can download it here : http://www.netzob.org/
A great example w/ ZeroAccess C&C protocol : http://www.netzob.org/documentations/presentations/netzob_29C3_2012.pdf
You can also take a look at CANAPE : http://www.contextis.com/research/tools/canape/
As for the automated approach, the file command is the classic recommendation. It's a tool which tries to guess the file type from the data contained in the file. It implements many of the well known file format signatures via libmagic. If you have a composite file you could try binwalk which will try to find files as subsets of a larger file.
If you want ...
I like file to determine the type of file from the header magic and Wotsit for standard file formats/documentation/reversed file format structures by other people.
Fileformat.info was suggested as another resource for those that don't like Wotsit or feel it's dated.
The numbers in the third column do increase over time, which is a good start. Let's check the differences between numbers on consecutive lines, to see if the progression is linear:
import re, sys, time
lines = sys.stdin.readlines()
def parse(l): return time.mktime(map(int,l[0:6]) + *3), int(l, 16)
stamps = [parse(re.split('[\n.: ]...
The Lets Solve the File Format Problem wiki covers quite a good number of obscure formats and is worth checking out if Wikipedia isn't doing the job for you. It's also a good place to contribute very detailed or obscure stuff that Wikipedia might deem out of scope.
You may find help or advice from the library and archival digital preservation community ...
You asked this a couple of months ago, but I'm going to answer anyway.
First, I fear that your decompression code is buggy. There are several horizontal lines of noise, and I'm pretty sure they're not supposed to be there.
Your A+B=C guess was correct. A tell-tale sign are the visible horizontal edges but hidden vertical edges. I think your mistake was ...
You don't state the purpose. If you want to fully understand the format and not just automate certain initial aspects of the analysis, it will be very hard to come up with a generic enough tool. This is what domain-specific languages and extensible tools (010 Editor, scripting languages) exist for. I'm not aware of any tools that would "do it all" at the ...
This is caused by a differences in the operating system's loader and the file format parsing code in the tool you are using. Malicious program authors often exploit differences between an executable file format's specification and how the file format is actually used by the loader in practice. If there are differences between the file format specification ...
I see filenames. That is an extremely important starting point - if I did not, I'd have to assume the file is encrypted, compressed, or does not use filenames at all, which are all harder to unpack.
For the moment, skip the header "BigFile" and the immedtaly following data and concentrate on these filenames alone.
If the filenames have different lengths, ...
Your best bet is Hachoir-Subfile. You can pass a file stream to Hachior-Subfile, it will search for all known embedded files and display the location. Some known formats it will calculate the size of the file. This makes it easy to carve out the files using dd. A helpful description of Hachoir-Subfile was left by one of the developers a couple weeks back in ...
I can't give you a specific solution, though I can tell you a tool to make reverse engineering a protocol easier.
Scapy is a python packet manipulation tool. One of the problems you have is, that wireshark doesn't know those packets. With Scapy its very easy to build and dissect strange/own packets. This will definetly help when you start to reverse ...
Here is the complete answer to everyone who may encounter compressed GIM file of simular compression.
Basicly the file starts like this:
[magic number 10 00 00 00] [Integer with uncompressed size of file]
After this the compressed file begin.
The compression basicly functions like this: (in terms of decompression)
-> Take the next 2 bytes.
-> Is the second ...
Concur with V and only throwing out some thoughts.
In general would say that one, (of course), has to clearly denote holes
as black or gray. As in "no idea" or "several possibilities".
Beside file format specifications openly distributed I find the style
used by RFC's to be one I adapt frequently. All depending on context
using such things as Augmented ...
From what I have recently gathered DxClient is designed as a client for DVR Netview technology. Just by looking at functionality of the DxClient, it is clear that it is more then just binary transfer of AVI formatted stream. I think, it is safe to assume, that rather proprietary transfer and control protocol is used. 2 frames that you provided is just not ...
Piece of cake: this is ZIP-compatible compressed compiled Java. Any familiar sequences of bytes near the header?
FA CE AF 0E 10 00 10 00 78 9C AC DD ..
.. those first few bytes look like Java's magic ID CA FE BA BE, but Googling that sequence doesn't yield any results, and it might be a red herring anyway. Next up: 10 00 10 00, which can be about anything (...
I published some tools on github which can do just that: https://github.com/nlitsme/pyidbutil and https://github.com/nlitsme/idbutil.
The first is written in python, the second in C++, both have similar functionality.
pyidbutil provides the most low level recovery options: using --pagedump you can dump each page in the file without the need of an intact ...
The precise BPMs are actually in the data files (filename).DAT (may the overall BPM is in the edb.. but I can't confirm)
So I have reversed both data files created by RekordBox:
/numbers are all big endian/
[tag] - 4 byte string
4byte - tag header size
4byte - segment size (including tag header)
(in multibit fields, msb-to-lsb (left-...
It is encrypted with AES so you will need the keys from windows.plist to decode.
The format is (all stored in big-endian):
0-3 magic ('NSCR' for PersistentUIRecord)
4-7 version (either '1000' or '0006')
8-11 NSWindowID (used to lookup 128-bit AES key stored in windows.plist)
12-15 record length (including from 0 to xxx)
It's certainly not a well known format. A quick glance at the file with a hex viewer shows that it mainly consists of records that all have similar, but not identical, size and layout; the very end of the file seems to be something different.
The first 2 Bytes - 047E - seem to be the number of records (1150).
Each of the records seems to start with 7 ...
NB: Problem is that such an endeavor is very specific to the file one is looking at. So a lot of common sense and presumably experience is required to deviate from the path outlined here when necessary.
Okay, you have numerous options, and all of us probably have kind of a recipe for this.
I for one would start with file (Windows version here).
Here's an explanation for what I think the individual symbols mean. I'm basing this around the presumption that a little selector is going through the cells, one by one.
\xFF = Null cell
\x05 = A string is following, with \xNumber coming after the string to define how far to displace the string from the selector's current position, if at all.