What is the structure of this database file from an old videogame? https://www.dropbox.com/s/ioe26pi3y1w3320/TMDATA1.S8?dl=0
Apologies in advance because fixing specific problems is not really the spirit of Stack Exchange, rather than creating a useful set of Q&As. If you can just point me in the right direction, I'd be really grateful. :-)
It's certainly not a well known format. A quick glance at the file with a hex viewer shows that it mainly consists of records that all have similar, but not identical, size and layout; the very end of the file seems to be something different.
The first 2 Bytes - 047E - seem to be the number of records (1150).
Each of the records seems to start with 7 strings that, in most cases, denote a country, adjective for the country, capital, and team name - i'd guess the save file is from a soccer game or similar.
For example, one of the record starts at 1B85 is 07 00 Algeria 07 00 Algeria 07 00 Algeria 07 00 Algeria 08 00 Algerian 07 00 Algiers 0F 00 Stade Olympique. Obviously, these are 2-bytes integers denoting a string length; the fact they are written as 07 00, not 00 07, tells you the file has a little endian structure, which may help identifying other numbers in the file.
Unfortunately, the records have different sizes, but if you take the different string lengths into account, they work out nicely. This is a list of the first record positions, their sizes, and the combined length if strings in them:
pos. size strings size w/o strings
0002 0597 005b 053C
0599 0582 0046 053C
0b1b 0582 0046 053C
109D 056D 0031 053C
160A 057B 003F 053C
1B85 0576 003A 053C
20FB 0569 002D 053C
2664 0570 0034 053C
2BD4 ........
Subtracting 0E for the 7 2-byte length markers results in 0x52E bytes that each record has behind the strings.
Next, i noticed that many of these bytes seem to be the same in many of the records. So i wrote a small perl program to read the file, and create a histogram to show which byte in which position occurs how often in the records:
#!/usr/bin/perl
open(IN, "<$ARGV[0]") or die "can't open $ARGV[0]: $!";
sysread(IN, $buffer, 2);
$nclubs=unpack("v", $buffer);
for (my $i=0; $i<$nclubs; $i++) {
printf("%3d ", $i);
# read strings
for (my $j=0; $j<7; $j++) {
sysread(IN, $buffer, 2);
my $length=unpack("v", $buffer);
sysread(IN, $buffer, $length);
print "|$buffer";
}
print "\n";
sysread(IN, $buffer, 0x52E);
for (my $j=0; $j<0x52E; $j++) {
my $byte=ord(substr($buffer, $j, 1));
$count[$j][$byte]++;
}
}
for (my $i=0; $i<0x52E; $i++) {
my $flag=0;
printf("%03x:", $i) unless $flag;
for (my $j=0; $j<256; $j++) {
if ($count[$i][$j]>0) {
$flag=1;
printf " 0x%02x(%dx)", $j, $count[$i][$j];
}
}
print "\n";
}
This results in (small extract):
314: 0x00(1150x)
315: 0x00(1150x)
316: 0x00(1134x) 0x6b(16x)
317: 0x00(1150x)
318: 0x00(1134x) 0x2c(16x)
319: 0x00(1100x) 0x6b(50x)
31a: 0x00(1150x)
31b: 0x00(1150x)
31c: 0x00(1083x) 0x6b(67x)
31d: 0x00(1150x)
31e: 0x00(1150x)
31f: 0x00(1100x) 0x6b(50x)
320: 0x00(1150x)
which means that bytes 314 and 315 are always zero, and byte 316 is 0 in most cases, but 0x6b (107) 16 times, i.e. for 16 teams. Byte 31c is 0x6b for 67 teams. You could now proceed to compare this to what happens when you load the save file into the game; is there anything that's special for exactly 16, or 67, of these 1150 teams? You could also look for the teams that have these values, and check what's special for these exact teams.
Investing a lot of time into this will possibly allow you to identify most bytes without actually decompiling the application. However, i'd expect you can identify only a few of the values this way, and you will need to do what Jason suggested if you really want to understand everything. On the other hand, if the team/capital/country names are the only thing you're interested in, my program will already output them for you.
The file does not have a well-known file header (as per http://www.garykessler.net/library/file_sigs.html). So unless it happens to be a SIMUL8 file (based on the file extension), you'd need to reverse engineer the video game's code to determine the file format.
