How to reverse engineer a database file format?

Question

It is an accounting database whose database file format is proprietary data. But the problem is the database is highly unreliable, corruption go unnoticed for years after which it becomes unbearably difficult to recover data, we maintain accounts of lots of companies, so reversing the format will be highly useful. Also, the software is odbc compliant which might be very helpul for reversing it.The software is partially open-source, i.e. except database engine everything is open-source(and fully customizable), though they have there own different language.
I have no experience in reversing. we would like to reverse it in such a way that we could write data in that format directly.

Answer 1

If there is no documentation, there's almost no chance to reverse the file format without reverse engineering the database engine. Especially writing the files will be a nightmare.

For example, the (very old) dBase II file format is quite easy to read, since the data itself is stored in a quite basic format, and for reading the database, you can ignore all the indexing stuff, you just read the data records. The format for index files is much more complicated, since they are (probably) binary b-trees, and you need to get the indexes right as well if you want to write stuff.

Now in your case, if "the database is highly unreliable, corruption go unnoticed for years" it seems that even the original designers of the software and the file format didn't get things right, at least in some edge cases. Which means you can't fix this without investing a massive amount of effort and money - we're probably speaking about multiple developers for several years.

If your software is ODBC compliant, you'll probably have a much better chance at success by plugging a different ODBC backend to your software frontend. This will fail if your software uses advanced features of your specific database (good luck trying to get oracle stored procedures running on a MS SQL server), but if your software doesn't use more than basic sql statements, this might at least work.