2

I'm trying to decode the file formats for Massive, FM8 and absynth VST synths. The file format is binary with several sections. Reason for this endeavour is to convert the above formats to general vst .fxp format for automated loading and rendering of presets through the vst api.

From experimenting with saving the files when changing synth parameters I've found out following facts about the format:

  • File starts out with length field
  • File contains some fixed length sections introduced by "DSIN", "hsin" markers
  • The vst binary chunk returned by effGetChunk vst sdk api calls are 99% similar to the .nmsv, .nabs, .nfm8 preset file content
  • changing a single parameter in the synth changes some bytes at several places (3-4) in the file
  • The main section of the preset file seems to be compressed, while synth parameters should be float values between 0 and 1 they don't seem to be written as floating points into the file - apart from unrecognizable binary values there are plaintext strings e.g. for user defined macros there seems to be some sort of compression applied to the raw data. Compression seems to reference previously encountered strings and reference these strings in later portions of the stream: If "THIS_IS_A_MACRO_NAME_ABC" is preceding "THIS_IS_A_MACRO_NAME_XYZ" in the stream the second string will be compressed to "[short sequence of bytes]_XYZ".
  • The files do not seem to contain any dictionary for compression which makes me think the dictionary must be stored somewhere else or there might not be a dictionary at all.

Could anyone help out here:

  • What compression scheme could be applied here?
  • Does anyone know about a similar format that has been successfully decoded?

Added an example file:

http://s000.tinyupload.com/index.php?file_id=08960658549599455274 contains a sample file. The string "TESTSTRING1" and "PREFIXTESTSTRING1SUFFIX" are contained in the uncompressed stream.

Shannon entropy is 5.84154 for the data chunk which is somewhere in the middle between english text and encrypted.

enter image description here

Here is an example which should demonstrate how the lenght field is computed: The string "TESTSTRING123" precedes the string "PREFIX...SUFFIX".

                             P  R  E  F  I  X  L1 L2 D     S  U  F  L1 D
----------------------------------------------------------------------------------------
PREFIXTESTSTRING123SUFFIX 05 50 52 45 46 49 58 E0 04 16 02 53 55 46 20 12 40 42 00
PREFIXTESTSTRING12SUFFIX  05 50 52 45 46 49 58 E0 03 16 02 53 55 46 20 11 40 41 00
PREFIXTESTSTRING1SUFFIX   05 50 52 45 46 49 58 E0 02 16 02 53 55 46 20 10 40 40 00 33 40
PREFIXTESTSTRINGSUFFIX    05 50 52 45 46 49 58 E0 01 16 02 53 55 46 20 0F 40 3F 00 33 40
PREFIXTESTSTRINSUFFIX     05 50 52 45 46 49 58 E0 00 16 02 53 55 46 20 0E 40 3E 00 33 40
PREFIXTESTSTRISUFFIX      05 50 52 45 46 49 58 C0    16 02 53 55 46 20 0D 40 3D 00 33 40
PREFIXTESTSTRSUFFIX       05 50 52 45 46 49 58 A0    16 02 53 55 46 20 0C 40 3C 00 33 40
PREFIXTESTSTSUFFIX        05 50 52 45 46 49 58 80    16 02 53 55 46 20 0B 40 3B 00 33 40
PREFIXTESTSSUFFIX         05 50 52 45 46 49 58 60    16 02 53 55 46 20 0A 40 3A 00 33 40
PREFIXTESTSUFFIX          05 50 52 45 46 49 58 60    16 01    55 46 20 09 40 39 00 33 40 
PREFIXTESSUFFIX           05 50 52 45 46 49 58 20    16 02 53 55 46 20 08 40 38 00 33 40
PREFIXTESUFFIX            05 50 52 45 46 49 58 20    16 01    55 46 20 07 40 37 00 33 40 
PREFIXTSUFFIX             09 50 52 45 46 49 58 54 53 55 46 20 06 40 36 00 33 40  
7
  • 1
    This sounds very like some form of LZSS compression. A similar recent question on here is reverseengineering.stackexchange.com/questions/21981/…
    – Ian Cook
    Commented Oct 30, 2019 at 10:24
  • Hi Ian, what makes you think this would be LZSS compression? Could it be LZ77, LZ78 or LZW? I think with huffman codeing the code would look less "readable". Is there no dictionary at all or maybe a fixed one stored somewhere else? Thanks alot
    – Martin
    Commented Oct 30, 2019 at 22:50
  • I've just come across LZSS more than the others. However, 11 consecutive plain text characters means it would have to be a different implementation to those I've seen before. Having some readable text in the compressed data does suggests it's a byte based method though. This likely rules out bit-stream based approaches e.g. anything using Huffman coding, or LZW with 12-bit or variable length codes.
    – Ian Cook
    Commented Oct 30, 2019 at 23:12
  • I don't really understand yet how length and distance are encoded here - I thought there are definitely some "control characters" (e.g. @ 40h) which appear more frequently than other characters.I'll try to give an examples with different length strings so this might shed some light.
    – Martin
    Commented Oct 30, 2019 at 23:32
  • It may be zlib. I had a quick look at my Massive in IDA and roughly in the neighbourhood where those DSIN and hsin markers are used and connecting stuff via RTTI names, I found zlib usage. According to the RTTI data they have some custom compressed stream class that appears to use zlib. Although the code also seems to check for 'zlib' or 'none' 4 byte markers which aren't in your sample preset. Commented Nov 1, 2019 at 2:26

3 Answers 3

3

I think the results of your well-designed tests help explain pretty much everything.

It is an LZ77 variant. Try the following scheme for decoding the length and offsets. (Encoded bytes are shown below as 8 binary bits.)

1st byte  2nd byte  3rd byte      token type  length    offset(*)
========  ========  ========      ==========  ======  ==============
000qqqqq     --        --     =>  literal     1 + Q         --      
001qqqqq  rrrrrrrr     --     =>  dictionary  3       (Q<<8) + R + 1
010qqqqq  rrrrrrrr     --     =>  dictionary  4       (Q<<8) + R + 1
011qqqqq  rrrrrrrr     --     =>  dictionary  5       (Q<<8) + R + 1
100qqqqq  rrrrrrrr     --     =>  dictionary  6       (Q<<8) + R + 1
101qqqqq  rrrrrrrr     --     =>  dictionary  7       (Q<<8) + R + 1
110qqqqq  rrrrrrrr     --     =>  dictionary  8       (Q<<8) + R + 1
111qqqqq  rrrrrrrr  ssssssss  =>  dictionary  9 + R   (Q<<8) + S + 1  

* the offsets are backwards in the decoded data from the current output position.

Edit The whole file from offset 00000410 onwards appears to be decompressible using the above encoding scheme. Doing so gives me the below. You can see that the end of my first excerpt has a number of 32 bit floating point values. The 2nd excerpt has your test string.

00000000:  F9 15 00 00 00 00 00 00 01 00 00 00 68 73 69 6E  ............hsin
00000010:  01 00 00 00 00 00 00 00 EC 2D F1 91 50 BC 4D 41  .........-..P.MA
00000020:  96 E1 F3 EA B1 70 A9 B9 18 00 00 00 00 00 00 00  .....p..........
00000030:  44 53 49 4E 01 00 00 00 01 00 00 00 01 00 00 00  DSIN............
00000040:  01 00 00 00 01 00 00 00 00 00 00 00 44 53 49 4E  ............DSIN
00000050:  6D 00 00 00 A5 15 00 00 00 00 00 00 01 00 00 00  m...............
00000060:  68 73 69 6E 01 00 00 00 00 00 00 00 85 50 7B 20  hsin.........P{ 
00000070:  83 65 FA 41 80 7B CA 65 CA E4 1D FA 75 15 00 00  .e.A.{.e....u...
00000080:  00 00 00 00 44 53 49 4E 6D 00 00 00 01 00 00 00  ....DSINm.......
00000090:  18 00 00 00 00 00 00 00 44 53 49 4E 01 00 00 00  ........DSIN....
000000A0:  01 00 00 00 01 00 00 00 01 00 00 00 01 00 00 00  ................
000000B0:  01 00 00 00 2D 15 00 00 00 00 00 00 18 00 00 00  ....-...........
000000C0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000000D0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000000E0:  00 00 00 3F 00 00 7D 3F 00 00 80 3F 00 00 80 3F  ...?..}?...?...?
000000F0:  00 00 00 3F BA 9E FF 3E 00 00 80 3F 00 00 80 3F  ...?...>...?...?
00000100:  00 00 80 3F 00 00 00 3F A6 30 00 3F 00 00 80 3F  ...?...?.0.?...?
00000110:  00 00 70 3F 00 00 80 3F 00 00 00 3F 00 00 80 3F  ..p?...?...?...?
00000120:  00 80 91 3E 00 00 00 3F 00 00 00 00 00 00 00 3F  ...>...?.......?
00000130:  00 00 00 3F 00 00 00 3F 00 00 00 3F 00 00 00 3F  ...?...?...?...?
00000140:  00 00 00 3F 00 40 41 3F 00 00 00 3F 00 00 00 00  ...?.@A?...?....
00000150:  00 80 99 3E 00 00 00 3F 00 00 00 3F 00 00 00 00  ...>...?...?....
00000160:  00 00 00 3F 00 00 00 3F 00 00 00 3F 00 00 00 3F  ...?...?...?...?
...
00000430:  00 00 00 00 00 01 00 00 00 07 00 00 00 00 00 00  ................
00000440:  00 00 00 00 00 01 01 00 00 00 00 C0 00 00 00 40  ...............@
00000450:  01 01 00 00 00 00 00 01 00 00 00 0B 00 00 00 54  ...............T
00000460:  45 53 54 53 54 52 49 4E 47 31 17 00 00 00 50 52  ESTSTRING1....PR
00000470:  45 46 49 58 54 45 53 54 53 54 52 49 4E 47 31 53  EFIXTESTSTRING1S
00000480:  55 46 46 49 58 01 00 00 00 33 01 00 00 00 34 01  UFFIX....3....4.
00000490:  00 00 00 35 01 00 00 00 36 01 00 00 00 37 01 00  ...5....6....7..
000004A0:  00 00 38 04 00 00 00 00 00 00 00 55 55 29 41 02  ..8........UU)A.
...
3
  • As a note: zlib's DEFLATE is a variation of LZ77 so this is one more hint towards zlib being used. Commented Nov 3, 2019 at 21:32
  • I use gchq.github.io/CyberChef to compress a sample string w DEFLATE - in the compressed string there are no ascii chars of the original text left. If i understand correctly this is due to huffman working on bit-level. I've looked for parameters to run DEFLATE without huffman but no luck. About the dictionary: to INFLATE above block I think I'd require a dictionary which i cannot spot in the preset file.
    – Martin
    Commented Nov 5, 2019 at 1:43
  • The dictionary is the previously decompressed data.
    – Ian Cook
    Commented Nov 6, 2019 at 6:45
1

I actually made some headway here on various files of this type, I've written a decompressor also for the LZ blocks.

I think hsin stands for 'header section in' and DSIN stands for 'data section in'. There's also 4kin, not sure what that is.

There are a few cases in certain files where an incorrect dictionary jump will be used - for example, offset 1, length 5. Could be my own error but the above algorithm seems to not account for this case.

I'm analysing the section blocks and they are a bit wacky. I've found length descriptors that seem to be completely ignored, amongst other things. So much of the file can be zeroed out no problems, with no noticeable effect (I think it's used for NIs indexer / library stuff).

Blocks are not actually fixed length, just look like they are (perhaps the first header is fixed length). There are length fields scattered throughout, but not for the first two blocks - they have 'bytes remaining' where the block length should be. If I zero out these fields, the file just reads without error. Weird.

Would you guys be interested in collaborating on a repository and sharing our findings?

5
  • Nice! I think a repository would help for any future endeavors. Actually I've given up on this because I didn't get any further than the state mentioned above. My goal was to partially understand the format and extract the blocks that are required to load the presets via vst api (which seems to have 99% overlap with the file format).
    – Martin
    Commented Sep 10, 2020 at 10:40
  • sure, look me up on telegram, @deathdisco - I've made a bit of progress deciphering the block format
    – monomadic
    Commented Sep 10, 2020 at 10:44
  • A length greater than the offset is valid in many LZ variants and not uncommon with an offset 1. If you think about the individual bytes being copied 1 at a time, it's effectively copying the last byte forward 1 byte, N times giving a repeated value. This means that you can't copy all length bytes out from the history and them write them. It has to be done 1 byte at a time.
    – Ian Cook
    Commented Sep 21, 2020 at 16:07
  • yup, it's actually a circular buffer (maybe it's called something else, but concept is just reaching backward). I have the decompressor totally working and the container blocks working - I think I just worked out data segments also. This is what worked in the end for the buffer stuff (excuse the mess for now): github.com/monomadic/ni-decompressor/blob/master/src/offset.rs
    – monomadic
    Commented Sep 23, 2020 at 1:45
  • @IanCook I'm really stuck on the data values themselves, so any help would be massively appreciated on those. I've also uploaded a kaitai grammar file to the repository if that helps. I really think we can crack this file format family wide open, and a lot of the work has been done but I'm stumped on a few areas. Would really appreciate any advice there :)
    – monomadic
    Commented Sep 23, 2020 at 14:53
0

I now have a working schematic and implementation for the NI file container, blocks, and decompressor. I still cannot interpret data blocks (basically variables in the packer), I'd love some help or discussion there. I understand basic things about DSIN blocks but not sure how the values relate to the data itself.

I've put my findings up at https://github.com/monomadic/ni-decompressor

This works with many NI files, not just massive, but I've been focusing on kontakt. I can also read monoliths but they are an entirely different format, but they look like they're easier.

I've pre-decompressed some compressed segments and dumped the decompressed section if you're just interested in those (though they won't read in kontakt, massive etc as they don't have the wrapper around them) in the /examples directory, with the extension .deflate

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.