I would suggest turning on each bit individually and see what tempo value is displayed. In other words, for each byte, test values 1, 2, 4, 8, 16, 32, and 64 while keeping the other bytes zero. Do the displayed tempo values double each time you double the byte value?
Because MIDI data bytes can only contain 7 bits, when an 8-bit byte needs to be sent in a System Exclusive message, manufacturers sometimes just put 4 bits in one data byte and the other 4 bits in the next data byte. So this is one kind of pattern to look for.
From the results you described, one hypothesis I have is that the turntable only cares about the lowest 4 bits in each byte. In other words, only values of 1, 2, 4, and 8 will have an effect on the tempo display.
byte5: 1 2 4 8
tempo: .01 .02 .04 .08
byte4: 1 2 4 8
tempo: .16 .32 .64 1.28
byte3: 1 2 4 8
tempo: 2.56 5.12 10.24 20.48
byte2: 1 2 4 8
tempo: 40.96 81.92 163.84 327.68
byte1: 1
tempo: 655.36
In this pattern, the smallest tempo value is .01. When the turntable displays the tempo, it appears to just drop the hundredths digit (.09 displays as .0 and .10 displays as .1) instead of actually rounding (.04 would display as .0 and .05 would display as .1).
On the other hand, some of the results you described suggest data byte values of 16, 32, and 64 may have some kind of effect? Maybe these values duplicate the effect of the 1, 2, and 4 values of the next byte to the left? Maybe there is some other pattern.