# reverse engineering file containing time series data

I have to decode a proprietary file format which mainly contains time series data. The only thing I was told was that the content is 'zipped', although I am not sure whether it is really zipped or some in-house compression method is used. Here is a snippet of the data:

789cecdd55d895d5bb057c26489774a7ac798f713feffbd2dd255d4a2b29
082220a1d24887020648194883747788a4840828024a4a238d347c13beef
60afb5f777fcac83e7372efedb8d27e3bae635e386d7b592c5cb9f375ebc
93f1e2c543bc4b09dac5eb302c7f3c5303a910835d32c2f6b579a41bc633
ab66d2945a4c47e95d36e52db44246647789178832c6b03792a0a574b2bb


I did a [byte] frequency analysis and it shows a uniform distribution. If the bytes are plot (with A.X.E.), a very homogeneous image is shown:

I've checked for the header of common compression algorithms but found none of them.

Any help is highly appreciated.

EDIT

Some of these file can be accessed on the following links: First file, Second file, Third file, Forth file

The executable which produced/validated these files is not available.

EDIT 2

Using binwalk -Me <filename> (github.com/devttys0/binwalk) it is possible to extract two zlib-ed files from each of the previous files.

Here's the header of one of the files:

0a002f220000180100002800e80364001f783c0001a0349c3574367136db
3582348932bb30dd30a12f2b2f542ff02e432f432ff62edd2f772f6a2f2e
2fbc2edb2d382dcc2cfe2c272d492d342d1d2d352d352d00000000000000
000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000

• Is the code generating or processing the file available to you? how many file samples can you collect? are you able to trigger any validation over modified files? can you share the entire file? more info would help here Sep 21 '16 at 10:49
• sharing the entire file would help immensely. also,what is there once you decompress the zlib'd blobs? Sep 21 '16 at 14:38
• Best way to analyse proprietary file format - is to analyze software that should receive and process this file format. I think you should start with RE of binary that able to create file with this file format, read it or convert from\to it. You have access to those? Sep 22 '16 at 11:20
• @DenisLaskov OP already mentioned he does not. "The executable which produced/validated these files is not available." izibe it would be nice of you to upload the uncompressed files, now that you've figured out the first layer. Sep 22 '16 at 19:39

It would help if I knew what the data represented, but I have determined the format of the files once unzipped. Each unzipped file has this structure:

short unk1;           // 10
short num_records;    // 8751  (8760 hours per year)
short unk2;           // 0
short num_samples;    // varies per file from 8 to 179
short unk3;           // 0
short unk4;           // 40
short unk5;           // 1000
short unk6;           // 100
int data_bytecount;   // varies per file (remaining byte count of the file)
Record records[num_records];


Where each Record is defined as: (defined as an 010Editor template)

typedef struct {
byte flag;                // 0 = no data; 1 = sample data follows
if(flag == 1)
short data[num_samples];
} Record;


For example, in the first file unzipped from "295", there are 8751 records with 179 samples each. Most records only have ~30 non-zero values. The samples for records[0] look like:

15185, 15185, 15060, 14968, 14719, 14583, ...


I suspect the data values are normalized floats, perhaps dividing them by 1000.0 or using 16384 as a zero reference for positive/negative numbers. I also suspect each file contains 1 year's worth of records with 1 per hour.

I'd be interested in knowing what the values represent ... just to have a complete picture.