Photoshop's Liquify filter allows you to distort images interactively in many different ways. When you have distorted an image to your liking, you can save the mesh that represents the transformation in a .msh file. This transformation can then be loaded and applied on other images. I need to reverse engineer .msh files even though I never never decoded a file format before. Below is everything I've figured out so far.
Previously, .msh files just contained a header and an uncompressed array of pixel displacement values. I know this because the format was reverse engineered in open source projects such as Paint.NET and graphics32 (see TVectorMap.LoadFromFile and SaveToFile). This resulted in very large file sizes, so Adobe eventually updated the format to include some kind of compression. The compression is very efficient (an 11.5 MB file was reduced to 430 kB when resaved in the new format), but I don't know if it is lossy or lossless. To my knowledge, these newer compressed .msh files have not been decoded, but this is what I need to do.
The new headers are very similar to the old headers, but I can't figure out what kind of compression has been used so I can't read the actual data. I googled some common compression formats and their signatures but didn't detect anything useful.
Is there a trick or utility that can help me figure out the type of compression? Any other thoughts that can lead me in the right direction?
If anyone wants to take a look, I created a few sample .msh files in vastly different sizes along with .png images that demonstrate the transformation being applied. I uploaded them to MEGA here (1.3 MB download):