Need help decoding bytes to number

Question

I've got a really really old file format from an accounting software, late '70s, DOS, QBasic or QuickBasic. Source code is lost, only .exe file remaining, file needs emulator.

The file formate seems to be mainly nullterminated strings and unsigned values, usually 2 bytes / 16 bits in Little Endian notation.

I have received some data and can read most of it, but there's a really strange encoding for the transaction value. I've collected all the data and posted it in a Google Sheet: https://docs.google.com/spreadsheets/d/1hV2cvtVP7YEMxwiEizNZYO89JF_zFhzJbUl9WqWTmx8/edit?usp=sharing (no sign-in required).

I have tried to decode the number.

• It's probably 8bytes/64bits, so the table contains the columns Y-AF for the bytes 0 to 7 (bits 0 to 63).
• These are the only datasets I know the values for.
• The expected value is noted in columns B and AM.
• Converted values are in Columns AI-AL (int8 BE / int8 LE / uint8 BE / uint8 LE), but none of those make any real sense to me
• Values are also plotted, see "Graphs" Tab

I have decoded the last byte (byte 7, column AF) and somewhat matched it with the expected value, IF the first (leftmost) bit is a 1. The (java) formula then is:

        final byte lastByte = s.Byte7;
        final byte testByte = (byte) (lastByte & 0b01111111);
        if (testByte == lastByte) continue; // does not start with 1-bit

        final double result = Math.pow(2, testByte - 1);

So in other words, if we have the byte 7 and lose its leading 1, it's v = 2^(byte7-1). This brings exact results for all values being of 2^x. Any other values are close, but I assume the other bytes need to be taken into account then, too.

Haven't gotten further yet. Perhaps it's an old standard of storing floating point values, but I've never seen this before, and it looks like it has nothing to do with the IEEE floating point numbers of Java, because I tried all conversions (LE/BE, bytes2float/double, signed/unsigned, exponential and logarithmic conversions).

I'll try Base64 and other not-8-bits-per-byte strategies next.

I'll post a bounty of 200 ASAP. (Update: A bounty of 100, as I'm new to this site and only have 101 reputation yet... sorry)

Update: Looks like this is simply the old Microsoft Binary Format. Now I'll write some converting algorithm to read it in Java.

Answer 1

Okay, I finally managed to convert it.

Usually a byte-wise conversion is way better than what I did here, but result are OK so far, and basically it's all the same anyway.

And yes, I'm not taking care of infinity or NaN or all the other exceptional values.

static public double fromMBF64(final byte[] msbin) {
    if (msbin == null) throw new IllegalArgumentException("msbin can not be null.");
    if (msbin.length != 8) throw new IllegalArgumentException("msbin has wrong size. Expected 8 bytes, but got " + msbin.length + " instead.");

    final int expBiased = JcUMath.toUnsigned(msbin[0]) - 128 - 1;
    final double exp = Math.pow(2, expBiased);
    final int sign = JcBits.testBit(msbin, 8) ? -1 : 1;
    double mantisse = 1;
    for (int i = 0; i < 55; i++) {
        final int pos = 9 + i;
        final boolean isSet = JcBits.testBit(msbin, pos);
        if (!isSet) continue;

        final double expVal = Math.pow(2, i + 1);
        final double off = 1 / expVal;
        mantisse += off;
    }
    return sign * exp * mantisse;
}

along with the utility functions

static public long toUnsigned(final int pValue, final long pMaxValue) {
    if (pValue >= 0) return pValue;
    return pValue + (2 * pMaxValue + 2);
}
static public short toUnsigned(final byte pValue) {
    return (short) toUnsigned(pValue, Byte.MAX_VALUE);
}

static public boolean testBit(final long pVariable, final int pTestPosition) {
    final long pattern = 1L << pTestPosition;
    return (pVariable & pattern) != 0;
}
static public boolean testBit(final byte[] pBytes, final int pIndex) {
    final int arrIndex = pIndex / 8;
    final byte b = pBytes[arrIndex];
    final int bit = 7 - pIndex % 8;
    return testBit(b, bit);
}

Note that the bytes have to be in Big Endian order. So if you're reading Little Endian, simply reverse the byte[]:

static public byte[] reverse(final byte[] pArray, final int pOffset, final int pLength) {
    if (pArray == null) return null;
    if (pLength < 1) return new byte[0];

    final byte[] ret = new byte[pLength];
    for (int i = 0; i < pLength; i++) {
        final byte t = pArray[pOffset + pLength - i - 1];
        ret[i] = t;
    }
    return ret;
}
static public byte[] reverse(final byte... pArray) {
    if (pArray == null) return null;
    return reverse(pArray, 0, pArray.length);
}

When I find the time I'll create a bit-shifting-based conversion, similar to this: https://gist.github.com/pmachapman/05094e12d0711745e3fd7093f7ab67cf, and post it in here.