How to convert hexadecimal values to 16-bit Unicode in IDA Pro 7.5?

Question

The context:

For a student project which goal is to make us learn reverse engineering and how to use IDA Pro, we have to reverse engineer the APT28 hospitality malware (as described in this blog post: https://blog.xpnsec.com/apt28-hospitality-malware-part-2/)

For this matter, we are provided with IDA Educational licenses (version 7.5) and the malicious Word file. After extracting the malware from the Word file, we can start reverse engineering the malware.

The problem:

As you can see in the blog post, it's possible (some way or the other) to make IDA decode the strings that are set in the load function (the following screenshot is from the blogpost):

However, when we tried to reproduce this, we couldn't find a way to do it. If I'm not mistaken, each mov instruction actually puts 2 characters encoded in 16-bit (widechar) unicode (UTF-8) into the specified address (found by applying an offset to ebp). We can confirm this by looking at the malware in Ghidra:

What I tried:

I tried to change the value of the ENCODING variable in ida.cfg so that auto-analysis recognizes UTF-8. (based on this: https://www.hex-rays.com/products/ida/news/7_0/docs/strlits/)

Based on this page IDA Convert to Unicode I tried 2 things:

• Select the value I wanted to convert, use Alt-A, and choose Unicode C-style (16 bits)
• Select all the addresses where the values are stored (in the Stack of load section), convert them to a String with A, then try to convert to Unicode C-style again.

When I try these 2 things, I end up with the following in the output window:

Command "SetStrlitStyle" failed.

End goal:

Ultimately, I can continue the work with Ghidra, but the goal is to actually learn new things about IDA and be able to use it correctly. There must be something I don't understand and I would love to increase my knowledge about this tool. We tried asking the teacher but he didn't know how to do it either.

Any help would be really appreciated.

Answer 1

The page you linked states that the ENCODING directive in the cfg file only applies to 1-byte-per-unit (1bpu) encodings, which this is not. UTF-16 is a 2bpu format.

The ENCODING directive is important because 1bpu characters are encoded differently depending on your culture. The values in the character table between 32 and 126 are usually the same, matching ASCII, but the low values and extended half (127-255) depend on the which alphabet encoding is being used, which is defined by the current culture. These encodings are defined in ISO 8859, which defines 14 different encodings for different cultures. ISO 8859-1 (Latin-1) is the most widely used, but other notable parts include 8859-5 (Cyrillic), 8859-6 (Arabic), and 8859-8 (Hebrew).

This then gets a little more complicated, because each implementation of these character sets (often referred to as codepages) is a little different. While ISO 8859-1 "Latin-1" is commonly considered to be the defining standard for Latin languages, you're significantly more likely to see Windows-1252 in practice, which is backward-compatible with ISO 8859-1 but includes many extended characters and defines encodings for control codes like \r and \n.

As you can imagine, this is effectively impossible to guess at the time of analysis. It's also tricky because UTF-8 is a variable length multibyte encoding that uses 1bpu a lot of the time for basic ASCII-range characters. The word cafe encodes to the same bytes in ASCII, ISO 8859-1, Windows-1252, and UTF-8. But add an accent on the e, to get café, and the encodings vary, with plain ASCII having no codepoint for é, ISO 8859-1 and Windows-1252 both encoding it as E9, and UTF-8 encoding it as C3A9. If you try to do this in reverse, ISO 8859-1 and Windows-1252 will turn C3A9 into é, which is completely valid but obviously not what was intended.

When you load a binary, IDA attempts to guess which 1bpu encoding was used based on the OS's current language settings. It uses that detected encoding (e.g. Windows-1252) in order to decode any 1bpu string literal bytes that it finds, into readable characters. If it can't guess, it defaults to UTF-8.

If the detected encoding isn't UTF-8, IDA then picks a culture based on the character encoding it picked. Each culture is defined in a .clt file and defines a number of common characters that exist outside of the common (ASCII-like) range for that particular culture. This is then used when detecting if a sequence of bytes might be a string literal. If the culture is incorrectly detected, IDA will still be able to identify ASCII strings, but strings with extended characters (e.g. café) might just show up as hexadecimal.

In short: for 1bpu strings, IDA detects the encoding in order to be able to turn bytes into readable strings, and (unless it's UTF-8) it uses a culture definition to detect string literals that contain extended codepoints.

The reason that UTF-8 is unaffected by these issues is that UTF-8 is just UTF-8 - the codepoints are the same everywhere regardless of the current culture or language.

IDA offers two config directives that can affect this process. The first is ENCODING, which overrides that first step and provides a specific character encoding to be used for decoding 1bpu characters. The second is CULTURE, which tells IDA which .clt file to use in order to detect which literals might be strings in the program.

Your situation is entirely unaffected by this, so setting ENCODING or CULTURE won't help you. UTF-16 is a 2bpu encoding that, like UTF-8, does not have different encodings depending on culture.

Unfortunately, I'm unclear as to why you're having that specific problem. The first thing to check is that you haven't selected the "character terminated" option in the string literal style dialog, as those individual instruction operands are not terminated strings. The only other thing I would suggest is attempting to interpret the strings as big-endian (UTF16-BE) in case IDA is attempting to convert the byte order incorrectly.

Answer 2

The main problem is that all of the string literals options or commands apply to string literals, i.e. a sequence of characters or code points. In your sample there is no complete literal; it’s built from pieces into a stack variable. For 8-bit (ASCII) strings you could use the R hot key to convert numbers into character constants (e.g. 65h -> ‘A’), however this does not work for UTF-16 fragments because of embedded zeroes.

I think the best you can do is collect the values copied onto the stack contiguously (e.g. ebp-14h, then ebp-10h etc.), reconstruct the corresponding array in another memory area or a script and then decode that as UTF-16.