I'm trying to reverse engineer a hash algorithm used by SharePoint 2016 in MSSCrawlUrl database table. I would like to avoid discussions about the idea itself (which I agree goes against SharePoint best practices), the answer is one word: legacy.
In short, that database table contains some URLs and their hashes. SharePoint somehow calculates a hash from values in AccessUrl column (which contains URLs), and puts the result into AccessHash column. The calculation is done by unmanaged (i.e. non-.Net) code.
So far I was able to figure out the following:
- AccessHash column definitely contains a CRC-32. In particular, the function is affine, i.e. hash(a ^ b ^ c) = hash(a) ^ hash(b) ^ hash(c) where ^ means XOR
- I could find a generating polynomial for the hash function (or I believe I could): 0x82608EDB. I used a technique outlined in this article: http://www.cosc.canterbury.ac.nz/greg.ewing/essays/CRC-Reverse-Engineering.html
After that I applied some brute force but couldn't figure out other algorithm parameters. The problem is a little bit more complicated because:
- I don’t know the actual encoding that SharePoint uses when it computes hashes. I tried several in my brute force code, but I'm not sure if I'm missing something.
- I don't have full control over the sample data, because a source string must be a valid URL (otherwise, SharePoint won't calculate a hash from it).
Here are some examples of URLs and their hashes:
AccessURL AccessHash http://a0 2141537348 http://a1 -1380154216 http://a2 -547891125 http://a4 981557741 http://a8 -169013482 http://aq -319117100 http://aa -55893049
More examples can be found in this file: https://bitbucket.org/johnnyjob/spreveng/src/master/CrcHack/Program.cs
The code which performs a brute force can be found here: https://bitbucket.org/johnnyjob/spreveng/src/master/
I'm open to any ideas, thanks in advance.