Help reversing a EDB database file for Pioneers Rekordbox software

Question

Pioneers Rekordbox software is a music management tool for DJs. One of its features is BPM detection for music files. Unfortunately it doesn't write this information to the BPM frame of the files ID3 tags, and instead keeps the detected BPM in it's own database files.

I'm writing a CLI tool to help me better manage my music, and one of the things I would like it to do, is extract the BPM data from the rekordbox database for each song.

As Guntram Blohm pointed at, the BPM is almost certainly not stored in the two ANLZ files. Instead it appears to be stored in the Rekordbox 'database.edb' file. I've uploaded an example edb file here which contains one track with the BPM '170'.

According to this forum thread the edb format is not an open format.

---

After a little more research it looks like it's a proprietary high-performance SQL database intended for use in embedded applications. (After EVEN MORE research, the database is called 'DeviceSQL'. Read the original authors Quora post about it). Doesn't seem like it is something that would be easy to reverse :(

For anyone else looking to extract the BPM information from their rekordbox library: It turns out there is a XML export that you can use. While not quite as automated as just reading the database file, it is a nice standard format!

---

Old question: Each track seems to have two files kept in the database (the database is just a directory tree of these files) for it. Here is the pair of files for an example track. The BPM was detected as 170 for the track, though I suspect it's storing it as a floating point or double as some other tracks can be detected at numbers like "169.96". Though looking for the double and floating point hex representations didn't yield any matches for me.

I see the files have what look like headers, for example "PPTH" followed by the file path. and "PQTZ", for the Rekordbox quantization feature. But I'm not too familiar with file formats, so I can't tell if it's using a standard file type. Or something more proprietary.

If anyone is interested in taking a look at the files and pointing me in the right direction it would be greatly appreciated! Right now I'm just trying to figure out how the BPM is stored.

---

Answer 1

The precise BPMs are actually in the data files (filename).DAT (may the overall BPM is in the edb.. but I can't confirm) So I have reversed both data files created by RekordBox:

file.DAT
--------
/numbers are all big endian/
[tag]   - 4 byte string
4byte   - tag header size 
4byte   - segment size (including tag header)
(in multibit fields, msb-to-lsb (left-to-right) is the general direction)
//////////////////////////////

PMAI    - main file descriptor
4byte   - head size (28)
4byte   - total file size
4byte   - ??? (1)
4byte   - ??? (65536)
4byte   - ??? (65536)
4byte   - ??? (0)

PPTH    - file path
4byte   - head size (16)
4byte   - tag size
4byte   - data length
data_bytes - file path in UTF16 (big endian) \0 terminated

PVBR    - VBR seek table
4byte   - head size (16)
4byte   - tag size (1620) (4*400+4)
4byte   - 0
  >entries>
    4byte   - file pos
  >last_entry>
    4byte   - ???

PQTZ    - Quantized time zones
4byte   - head size (24)
4byte   - tag size
4byte   - 0
4byte   - ??? (524288=0x80000)
4byte   - number of entries
  >entries>
    2byte   - beat phase (1-2-3-4)
    2byte   - bpm*100
    4byte   - time index (msec)

PWAV    - Low resolution Wave display data (5+3bit)
4byte   - head size (20)
4byte   - tag size (420)
4byte   - data size (400)
4byte   - ??? (65536)
  >entries>
     3bit   - color index
     5bit   - height

PWV2    - Lowest resolution Wave display data (4bit)
4byte   - head size (20)
4byte   - tag size (120)
4byte   - data size (100)
4byte   - ??? (65536)
  >entries>
      4bit  - 0
      4bit  - height

PCOB    - CUE Object    ///first PCOB for hot cues, second PCOB for memory
                        ///only generated for USB storage, 
                        ///otherwise contains only dummy data and actual cue data stored in the edb 
4byte   - head size (24)
4byte   - tag size
4byte   - hotCUE? (1=hot cue, 0=memory)
4byte   - number of cue points
4byte   - memories (-1= hot cue)
  >entry tags>
    PCPT    - CUE Point
    4byte   - head size (28)
    4byte   - tag size (56)
    4byte   - hot cue no#, 0 otherwise
    4byte   - active (0=inactive / 4=active)
    4byte   - (65536)
    4byte   - ???? -----point type: 0xffff ffff = hot cue ////  memory first: 0xffff xxxxx ----- memory last: 0x xxxx ffff
      >datas>
        1byte   - cue type 1 = single / 2 = loop
        1byte   - 0
        2byte   - ??? (1000)
        4byte   - start time (msec)
        4byte   - loop end (-1 if not used)
        16byte  - 0



file.EXT
--------
PMAI    - main file descriptor
4byte   - head size (28)
4byte   - total file size
4byte   - ??? (1)
4byte   - ??? (65536)
4byte   - ??? (65536)
4byte   - ??? (0)

PPTH    - file path
4byte   - head size (16)
4byte   - tag size
4byte   - data length
data_bytes - file path in UTF16 (big endian) \0 terminated

PWV3    - High resolution Wave display data
4byte   - head size (24)
4byte   - tag size
4byte   - ??? (1)
4byte   - data size
4byte   - ??? (0x0096 0000)
  >entries>
      3bit  - color 
      5bit  - height

When I was reversing, there were no PKEY in the files, so I don't know what it is for (and only seems to have 0 in it on the PC)

So the BPM values are stored in the PQTZ tag (in dynamic mode, you can have different BPMs during the same song, so it makes sense)

Answer 2

The file format seems, as you found out, to consist of headers that have a tag. Each of these headers seems to be 16+ byte, with 4 bytes for the tag, 4 bytes for the length of the header, 4 bytes for the size of header + data, and 4 bytes that i'm not sure about. Unfortunately, these length bytes are big endian, which made me think the bpm could be stored in big endian IEEE float as well, which could be the reason you didn't find anything.

The first header, PMAI. seems to be some kind of envelope (its length field is the size of the file itself), the rest of the headers seem to various forms of data content.

I wrote a small program to dump the section names and lengths (please don't use it as an example for good style!):

    #include <stdio.h>
    #include <arpa/inet.h>

    int main(int argc, char **argv) {
            analyze(argv[1]);
    }

    int analyze(char *filename) {
            FILE *fp;
            struct {
                    char tag[4];
                    int  x0;
                    int  x1;
                    int  x2;
            }  header;
            int length;

            if ((fp=fopen(filename, "rb"))==NULL) {
                    perror(filename); return;
            }
            while (fread(&header, sizeof header, 1, fp)) {
                    header.x0=ntohl(header.x0);
                    header.x1=ntohl(header.x1);
                    header.x2=ntohl(header.x2);
                    printf("%04x %4.4s: %08x (%06d) | %08x (%06d) | %08x (%06d)\n",
                            ftell(fp)-sizeof(header),
                            header.tag,
                            header.x0, header.x0,
                            header.x1, header.x1,
                            header.x2, header.x2
                    );
                    if (!memcmp(header.tag, "PMAI", 4)) {   // outer container
                            length=header.x0;               // just skip header
                    } else if (!memcmp(header.tag, "PPTH", 4)) {
                            int i;
                            for (i=0; i<header.x2; i+=2) {
                                    getc(fp);
                                    putchar(getc(fp));
                            }
                            putchar('\n');
                            continue;
                    } else {
                            length=header.x1;               // else skip data
                    }
                    fseek(fp, length-sizeof(header), 1);
            }
    }

which produces the following output:

    $ ./sections ANLZ0000.DAT
    0000 PMAI: 0000001c (000028) | 000028fc (010492) | 00000001 (000001)
    001c PPTH: 00000010 (000016) | 00000100 (000256) | 000000f0 (000240)
    E:\music\247 Hardcore\[+singles]\[247HC055] [12B] Al Storm Ft. Malaya - Everytime We Say Goodbye (Technikore Remix).mp3
    011c PVBR: 00000010 (000016) | 00000654 (001620) | 00000000 (000000)
    0770 PQTZ: 00000018 (000024) | 00001f40 (008000) | 00000000 (000000)
    26b0 PWAV: 00000014 (000020) | 000001a4 (000420) | 00000190 (000400)
    2854 PWV2: 00000014 (000020) | 00000078 (000120) | 00000064 (000100)
    28cc PCOB: 00000018 (000024) | 00000018 (000024) | 00000001 (000001)
    28e4 PCOB: 00000018 (000024) | 00000018 (000024) | 00000000 (000000)

    $ ./sections ANLZ0000.EXT
    0000 PMAI: 0000001c (000028) | 0000cf56 (053078) | 00000001 (000001)
    001c PPTH: 00000010 (000016) | 00000100 (000256) | 000000f0 (000240)
    E:\music\247 Hardcore\[+singles]\[247HC055] [12B] Al Storm Ft. Malaya - Everytime We Say Goodbye (Technikore Remix).mp3
    011c PWV3: 00000018 (000024) | 0000ce26 (052774) | 00000001 (000001)
    cf42 PKEY: 00000014 (000020) | 00000014 (000020) | 0000000c (000012)

So, PMAI is the container. PPTH is the name of the MP3 file. PVBR is probably information about variable bit rate, PQTZ the quantization, and PWAV, PWV2 and PWV3 various wave forms. Which leaves only PCOB and PKEY to possibly contain the bitrate. Unfortunately, if you look at the hex dump of these:

    000028c0  xx xx xx xx xx xx xx xx xx xx xx xx 50 43 4f 42   ............PCOB
    000028d0  00 00 00 18 00 00 00 18 00 00 00 01 00 00 00 00   ................
    000028e0  ff ff ff ff 50 43 4f 42 00 00 00 18 00 00 00 18   ....PCOB........
    000028f0  00 00 00 00 00 00 00 00 ff ff ff ff               ............

    0000cf40  xx xx 50 4b 45 59 00 00 00 14 00 00 00 14 00 00   ..PKEY..........
    0000cf50  00 0c 00 00 00 00

it seems that PCOB contains 00 00 00 00 ff ff ff ff, and PKEY has 00 00 00 00. None of these look like they could mean 170.

This article says 'If rekordbox crashes on startup, rename database.backup.edb to database.edb, if it still crashes, remove all the datafiles'. Since the BPM don't seem to be stored in the ANLZ.* files - do you have a database.edb as well? Could the BPM be stored there?

Answer 3

The EDB format is used by Microsoft's Extensible Storage Engine (ESE) to provide a storage back-end to a number of applications and services (Exchange, Active Directory, Desktop Search, Windows Live Mail, etc.)

Although the EDB format itself is not documented, it is well supported through a Windows API.

You can also access the contents of an EDB file through libesedb.

If all you want to do is look through the contents of an EDB file in a human-readable manner, the EseDbViewer tool is very good.