How to identify in assembly base 64 encode/decode with Ida. Is there any magic constant that I can search? Or maybe a plugin that find that?
FindCrypt2 Ida plugin not find that
One of the ways that I have taught (see Day 1 Part 11) to find a base64 alphabet is to look for a sequence of 64 bytes with an Shannon entropy of 6 (on a scale of 1-8). Beware that in addition to Base64 alphabets, that will also find other high-entropy data, such as compressed or encrypted code or strings, as well as encryption constants.
For your standard Base64 alphabet, using IDA, press alt-B, then enter (including the double-quotes) "ABCDEFGHIJKLMNOPQRSTUV" or some other subset of the standard alphabet. I recommend also selecting the "find all" option, which makes it easier to see and skip false matches.
The common base64 key contains
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/. However, remember that IDA Pro will convert ASCII text for you, so you should have no reason to go looking through the raw bytes of the file in hex. In fact, even a hex editor such as HxD or 010 Editor will convert the string data from hex bytes to chars and display them on the right side of the screen. See below an example in 010 Editor of such key:
And in IDA Pro IDA View window looking at segments:
Nevertheless, if you need to go looking for a constant, use the ASCII table and convert the above string into bytes manually. E.G. 'A' is 41h, 'B' is 42h, so if we were looking for AB, we'd be looking for 41 42. Don't forget to take into account the endianness of the target architecture of the software system you are working with; if the bytes are stored on disk in little endian for the file format, than the base64 key's bytes will be encoded backwards and you will need to search for it as such.
One caveat here is that the aforementioned Base64 key is the standard Base64 key. You may need to search for modified Base64 keys, meaning that instead of
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ we have
FGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ABCDE. You can see that with this modified Base64 key, we've simply rotated the key to the left 5 times so that if a decoding algorithm were to attempt to decode this as standard Base64, it would not work.