0

While reverse engineering a database middleware. This is presumably encoding a list of double or float values with Base64 encoding and then compressing it.

Via zlib.decompress() I was able to decompress it, but I got a string of presumably multiple base64-strings. I can see that it consists of more than one, because it contains multiple =. But afaik, these do not always mark the end of a b64-encoded string, because one can also end without a =.

This is a excerpt of one field:

b'<\x12u<\xf4\x808=\x95\xf0U=\x9fse;\xef\xbe\xf2=A\xd1K=\x1fB\x99=\x95\x11\xa3=\xb8\xcd\xbd<\xd2\xfaL=\x1d\x80U=\xd6:\x1e=\xdcp\xcd=1\xd2\xe1=\t\x01\x8e<\x85\xa8\x16>8\xb4\xa7>2\xf4\x11=>\x03\x9b<\xdfA\x9b>%>a>\xcf\x9a\x05>Ie\x1c>\t@Y<\xdfC\xe2=\xf8\'\xb0=zpa=\x8e\xe8\xde<\xc8\xcby=\x88\xfe\xb6=\xb8Uv=\xd5\xe3\xee=q\xef|<B\xe1\x1f=%\xfe\x85=\x90_\x04=p\x9e\xbd=\x89og=\x96\x88\x87<\xa2\x9c\x84=\x969\xaf=\xab\x84^<\xef\x81\xf6<T\x7f\xf4<\x85\xd6\x86<\x80Q\x93<\xb4\xf9\x00<\xfc&s<\xb9q\x1b<\xd3\xd8\xa0<4\xe9\xc3=\x86a\xb4=\xd5s_=\xc8\xb1==\xc24\xca=~\xd3\xe8=^7\xa5=e\xa3-=\x07?4<\xd5HJ=='

As an amateur on this field, I am very unsure where to start. From the documentation I know, that it's a b64-encoded list, but I don't know how to use it. Obviously, when decoded, it will consist of a binary format that in some computer language represents a list of floats.

Any tips on how to continue working on this problem? Unfortunately I have no access to the software currently, the only thing I have is this data structure.

Sorry for my amateur questions, and thanks in advance for any tips!

1
  • 2
    A Base64 string will not have \x12.....\xf4 etc hope i am not understanding wrong here
    – blabb
    Oct 26 '21 at 17:49
3

The data you have here is not base64 encoded, as that would only have letters, numbers, +, /, and (as you mentioned) =. The \x escape code in the string indicates bytes that are non-printable or outside the ASCII range.

The regular pattern of = or near-= values (;, <, =, >) every four bytes suggests that this is a simple array of 4-byte little-endian fields with values relatively close to each other. Here is what it looks like as a 4-byte (single precision) float array.

>>> data = b'<\x12u<\xf4\x808=\x95\xf0U=\x9fse;\xef\xbe\xf2=A\xd1K=...'
>>> struct.unpack_from('<{}f'.format(str(len(data)//4)), data)
(0.014957960695028305, 0.045044854283332825, 0.05223139002919197, 0.003501154249534011, 0.11852823942899704, 0.049760106950998306, 0.07483314722776413, 0.07962337881326675, 0.0231693834066391, 0.05004388839006424, [...])

Does this look like the range of values you are expecting?

1
  • That's exactly what I was looking for, yes! Thanks! Quick noob question: How do you acquire this knowledge about the patterns? Any specific books, or is it just experience? Nov 24 '21 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.