12

I want to open finnish sports league "data file" used for bookkeeping. It includes all statistics for few decade, so it's interesting data file.

The file is here: http://www.bittilahde.fi/Tietokanta.dat (Database.dat in english)

The book keeping program is here: http://www.pesistulokset.fi/Kirjaus504.exe

What I've found out:

  • The histogram of database file is completely flat
  • There's no header I could recognize in database file
  • The .exe is compiled with Delphi 4
  • I can find some data structures with IDR, but cannot figure out how uncompress the file.

What could be the next step?

  • Did you try binwalk? – EfForEffort Jun 15 '13 at 17:27
  • @bueno No, I'm new at this so I've never heard of it.Do you mena this: code.google.com/p/binwalk – Harriv Jun 15 '13 at 17:38
  • yup, it crawls the file looking for all kinds of different file types. It requires python and libmagic, I think. it's good stuff. – EfForEffort Jun 15 '13 at 20:00
  • @bueno Looks like great program, but in this case it doesn't tell me anything useful. Entropy graph gives out flat 1.0 for whole file. – Harriv Jun 15 '13 at 20:26
  • 2
    I added binwalk as an answer, because it may help other people having a similar question. Some additional information about entropy graph and compression can be found here: devttys0.com/2013/06/… – samuirai Jun 19 '13 at 13:39
16

Looking at it in OllyDbg it looks like a heavy task. Looks like a custom database with encrypted and (custom?) compressed data. This or the like would usually be the case in such applications. A flat file with structured data is not part of this one.

Anyhow. As a starter:

A quick check after trying out some general compression tools like 7z or binwalk, (have not tested it), can be to use ProcMon from Sysinternals. Start ProcMon, then your application and set filter on the application in ProcMon. You quickly find that:

In short it reads in chunks of varying size, but for main data processing it reads chunks of 16384 bytes. The process in steps:

  1. Generate seed map of 256 integers. (Done once at application start.)
  2. Loop:
    2.1 Read 16384 bytes into buffer from .dat file.
    2.2 XOR routine on buffer using offset and last four bytes of buffer as base.
    2.3 Checksum on XOR'ed buffer using seed map from step 1.
    2.4 Parse buffer and read out data.

The application also reads same chunks multiple times.


2.1:

Example:

013D0010  D4 9E BE BF 1C 1C 0B D4 C5 E7 11 B5 09 48 87 FA  Ôž¾¿ÔÅçµ.H‡ú
013D0020  29 4C 03 C9 DE 4A 2B 71 74 7F D2 48 E7 13 94 4E  )LÉÞJ+qtÒHç”N
...
013D3FF0  6A D1 55 92 E2 16 60 53 69 89 86 7D D9 D8 10 BC  jÑU’â`Si‰†}Ùؼ
013D4000  90 F3 D1 48 28 47 34 EC 39 36 EC 4D 69 2A 7D E5  óÑH(G4ì96ìMi*}å
                                             |_____._____|
                                                   | 
                         Last DWORD aka checksum --+

Steps and details in order of discovery:

Split the .dat file in chunks of 16384 bytes and also generate a hex-dump of each file for easy search and comparison. To be honest I use Linux for this part with dd, xxd -ps, grep, diff etc.

Start OllyDbg, open the application, locate CreateFile and set breakpoint:

00401220   $-FF25 18825000  JMP DWORD PTR DS:[<&kernel32.CreateFileA>;  kernel32.CreateFileA

Press F9 until filename (in EAX) is .dat file. Set/enable breakpoint on ReadFile. F9 and when read is done start stepping and looking at what is done.


Looking at it:

2.2:

After read it first modify the buffer by using offset as "magic" starting at:

0045F5EC  /$ 53   PUSH EBX     ;  ALGO 2: XOR algorithm - post file read.
...
0045F6B6  \. C3   RETN         ;  ALGO 2: RETN

At least two of the actions taken seems to be libj_randl1() and libj_randl2(). (This would be step 2.2 in list above.)

Simplified:

edx = memory address of buffer
ecx = offset / 0x4000
edi = edx
ebx = ecx * 0x9b9
esi = last dword of buffer & 0x7fffffff
ecx = 0

i = 0;
while (i < 0x3ffc) { /* size of buffer - 4 */
    manipulate buffer
}

The whole routine translated to C code:

int xor_buf(uint8_t *buf, long offset, long buf_size)
{
    int32_t eax;
    int32_t ebx;
    int32_t esi;
    long i;

    buf_size -= 4;

    ebx = (offset / 0x4000) * 0x9b9;
    /* Intel int 32 */
    esi = (
        (buf[buf_size + 3] << 24) |
        (buf[buf_size + 2] << 16) |
        (buf[buf_size + 1] <<  8) |
         buf[buf_size + 0]
        ) & 0x7fffffff;

    for (i = 0; i < buf_size /*0x3ffc*/; ++i) {
        /* libj_randl2(sn) Ref. link above. */
        ebx = ((ebx % 0x0d1a4) * 0x9c4e) - ((ebx / 0x0d1a4) * 0x2fb3);

        if (ebx < 0) {
            ebx += 0x7fffffab;
        }

        /* libj_randl1(sn) Ref. link above. */
        esi = ((esi % 0x0ce26) * 0x9ef4) - ((esi / 0x0ce26) * 0x0ecf);

        if (esi < 0) {
            esi += 0x7fffff07;
        }

        eax = ebx - 0x7fffffab + esi;

        if (eax < 1) {
            eax += 0x7fffffaa;
        }

        /* Modify three next bytes. */    
        buf[i] ^= (eax >> 0x03) & 0xff;

        if (++i <= buf_size) {
            buf[i] ^= (eax >> 0x0d) & 0xff;
        }
        if (++i <= buf_size) {
            buf[i] ^= (eax >> 0x17) & 0xff;
        }
    }

    return 0;
}

Then a checksum is generated of the resulting buffer, (minus last dword), and checked against last dword. Here it uses a buffer from BSS segment that is generated upon startup, step 1. from list above. (Offset 0x00505000 + 0x894 and using a region of 4 * 0x100 as it is 256 32 bit integers). This seed map seems to be constant (never re-generated / changed) and can be skipped if one do not want to validate the buffer.

1.

Code point in disassembly (Comments mine.):

0045E614 . 53   PUSH EBX           ;  ALGO 1: GENERATE CHECKSUM MAGICK BSS
...
0045E672 . C3   RETN               ;  ALGO 1: RETN

The code for the BSS numbers can simplified be written in C as e.g.:

int eax;    /* INT NR 1, next generated number to save */
int i, j;

unsigned int bss[0x100] = {0};  /* offset 00505894 */

for (i = 0; i < 0x100; ++i) {
    eax = i << 0x18;
    for (j = 0; j < 8; ++j) {
        if (eax & 0x80000000) {
            eax = (eax + eax) ^ 0x4c11db7;
        } else {
            eax <<= 1;
        }
    }
    bss[i] = eax;
}

2.3:

That bss int array is used on the manipulated buffer to generate a checksum that should be equal to the last integer in the 16384 bytes read from file. (Last dword, the one skipped in checksum routine and XOR'ing.). This would be step 2.3 in list above.

unsigned char *buf = manipulated file buffer;
unsigned char *bss = memory dump 0x00505894 - 0x00505C90, or from code above

eax = 0x13d0010;  /* Memory location for buffer. */
edx = 0x3ffc;     /* Size of buffer - 4 bytes (checksum). */

...

At exit ecx is equal to checksum.

Code point in disassembly (Comments mine.):

0045E5A8  /$ 53  PUSH EBX    ;  ALGO 3: CALCULATE CHECKSUM AFTER ALGORITHM 2
...
0045E5E0  \. C3  RETN        ;  ALGO 3: RETN (EAX=CHECKSUM == BUFFER LAST 4 BYTES)

Shortened to a C routine it could be something like:

int32_t checksum(int32_t map[0x100], uint8_t *buf, long len)
{
    int i;
    int32_t k, cs = 0;

    for (i = 0; i < len; ++i) {
        k = (cs >> 0x18) & 0xff;
        cs = map[buf[i] ^ k] ^ (cs << 0x08);
    }

    return cs;
}

It is checked to be OK and then checksum in buffer is set as: two least significant bytes = 0, two most significant bytes are set to some number (chunk number in file or read number, (starting from 0)).

0045F9BF   . C680 FC3F0000 >MOV BYTE PTR DS:[EAX+3FFC],0     ;  Set two lower bytes of checksum in dat buf to 0
0045F9C6   . C680 FD3F0000 >MOV BYTE PTR DS:[EAX+3FFD],0     ;  follows previous
0045F9CD   . 66:8B4D F8     MOV CX,WORD PTR SS:[EBP-8]       ;  Set CX to stack pointer value of addr EBP - 8 (counter of sorts)
0045F9D1   . 66:8988 FE3F00>MOV WORD PTR DS:[EAX+3FFE],CX    ;  Set .dat buffer higher bytes like CX.

Now after all this is done the actual copying of data starts with even more algorithms. Here the real work starts. Identifying data types, structures, where and what etc. Found some routines that extracted names etc. But everything being Finnish didn't help on making it easier to grasp ;).

The data above could be a start.

Some breakpoints that might be of interest to begin with:

Breakpoints
Address    Module     Active    Disassembly  Comment
0045E5A8   Kirjaus5   Disabled  PUSH EBX     ALGO 3: CALCULATE CHECKSUM AFTER ALGORITHM 2
0045E5E0   Kirjaus5   Disabled  RETN         ALGO 3: RETN (EAX=CHECKSUM == BUFFER LAST 4 BYTES)
0045E614   Kirjaus5   Disabled  PUSH EBX     ALGO 1: GENERATE CHECKSUM MAGIC BSS
0045E672   Kirjaus5   Disabled  RETN         ALGO 1: RETN
0045F5EC   Kirjaus5   Disabled  PUSH EBX     ALGO 2: FILE POST XOR READ ALGORITHM
0045F6B6   Kirjaus5   Disabled  RETN         ALGO 2: RETN

Some notes:

Keep a backup of the .dat file you are working with. If you abort the application the file often gets corrupted as it, as noted by @blabb, write data back to file. The .dat file also seem to be live so a new download of it would result in different data.

  • 1
    One question: how did you create those C code snippets? By hand or using some tools? – Harriv Jul 24 '13 at 18:47
  • 2
    @Harriv: By hand. One way, when routines are not to long, could be to create a variable for each register (in use) etc. and almost make a blueprint in C, then you see what is what and can make functions and loops from it. After a while, and with exercise, it becomes easier to recognize what is what and how to write it more compact first time around … – Runium Jul 30 '13 at 15:18
5

the .dat file you posted is written back every time it is accessed by your app constant size of 9.6 MB (0x267 * 0x4000 ) = 0x99c0000 bytes

4000 bytes are accessed on each successive reads 267 times ReadFile is Done each of the 4000 bytes is xorred using a routine (custom ??) and then check summed for (int i = 0; i< 0x267 ;i++) { checksum = *(DWORD *) (FilePointer)(0x3ffc + i) } so on a dead dat file opened via hexeditor the dwords 0x3ffc,0x7ffc ...... contains the latest checksum

you might have to reverse engineer the xorring routine and checksum routine

analyse all the callstacks in the log file generated by windbg with this break point

C:\>cdb -c ".logopen kir.txt;bu ntdll!NtReadFile \".if ( poi(esp+0x1c) != 0x400 0) {gc} .else {kb;g }\"" Kirjaus504.exe

  • Great, thanks. I just assumed it was compressed.. – Harriv Jun 16 '13 at 7:37
4

Well from this point on, your best bet would be to reverse the program's database loading functions since the database might be a custom one. Place a break point on CreateFile/OpenFile APIs when it loads the database and then look for the database manipulation functions to see how it extracts the contents. Couldn't try this on the program you posted since the user interface is in Finnish.

4

binwalk is a firmware analysis tool. It comes with some default file signatures called magic. There is a magic file called compressed, which scans for common compression signatures. It shows a lot of false positives, but it may find some common compression types and it should also be capable of unpacking them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.