Any ideas on how to decode Photoshop Liquify .msh files?

Question

Photoshop's Liquify filter allows you to distort images interactively in many different ways. When you have distorted an image to your liking, you can save the mesh that represents the transformation in a .msh file. This transformation can then be loaded and applied on other images. I need to reverse engineer .msh files even though I never never decoded a file format before. Below is everything I've figured out so far.

Previously, .msh files just contained a header and an uncompressed array of pixel displacement values. I know this because the format was reverse engineered in open source projects such as Paint.NET and graphics32 (see TVectorMap.LoadFromFile and SaveToFile). This resulted in very large file sizes, so Adobe eventually updated the format to include some kind of compression. The compression is very efficient (an 11.5 MB file was reduced to 430 kB when resaved in the new format), but I don't know if it is lossy or lossless. To my knowledge, these newer compressed .msh files have not been decoded, but this is what I need to do.

The new headers are very similar to the old headers, but I can't figure out what kind of compression has been used so I can't read the actual data. I googled some common compression formats and their signatures but didn't detect anything useful.

Is there a trick or utility that can help me figure out the type of compression? Any other thoughts that can lead me in the right direction?

If anyone wants to take a look, I created a few sample .msh files in vastly different sizes along with .png images that demonstrate the transformation being applied. I uploaded them to MEGA here (1.3 MB download):
https://mega.nz/#!We5QkARI!2-qD_qK2e8ADnYJcT578x9RxXo4uvBH0rXeDPeR-pRU

Answer 1

Here's a description of the .msh format on Adobe Photoshop CC 2019, at least of what I've been able to glean.

The format uses run-length encoding to encode the offset coordinates of the mesh cells and compress empty cells along the way.

The format is little endian. The header is 64 bytes. At 0x10 and 0x14 are the mesh width and height, 32-bit integers. At 0x28 and 0x2c are the image height and width, 32-bit integers. The same image height and width are also found at 0x38 and 0x3c. The mesh size seems to always be 1/4th of the image size, so you get 4 pixels mesh cell.

The encoded grid starts at 0x40. Every row of mesh cells is encoded separately from the other rows.

Every row starts with the number of contiguous zero-offset cells, left-to-right, 32-bit integer. If the leftmost cell is non-zero then this number is zero. Then, if the row is incomplete, the number N of contiguous non-zero cells comes. Then come N pairs of 32-bit floats. If the row is incomplete, comes the number of contiguous zero-offset cells and so on until the row has been completely encoded. Then comes the next row with the number of zero-offset cells and so on until all rows have been encoded.

In the example above we have a 32x32 image with an 8x8 mesh. Starting at 0x40 we have:

• (row 1 starts here) 4 zero-offset cells
• 1 non-zero cell, 2 floats
• 3 zero-offset cells (the row is now complete)
• (the pattern repeats for two more rows until 0x7c)
• (row 4 starts here) 0 zero-offset cells
• 1 non-zero cell, 2 floats
• 3 zero-offset cells
• 1 non-zero cell, 2 floats
• 3 zero-offset cells (the row is now complete)

...and so on.

The encoded float pairs seem to be the X and Y offsets from the current pixel to the pixel from which to sample the value. Not sure if the magnitude is a distance between pixels or distance between cells, though.