1

I've seen this multiple times, through various apps and snooping of hex values. Character strings but every other value is actually a null byte. This particular example is with API Monitor, but I'm almost positive I've seen it through hexdumps of Windows or DOS executables.

CreateFileW Raw data

My first thought might be Unicode or some other sort of wide character standard. But Unicode formats (especially Uni16) are variable width.

4
  • No. In Windows "Unicode" refers to a fixed-with character type where every characters has 16 bits (2 bytes). Win32 API methods that end with a W (like CreateFileW) usually require those strings. learn.microsoft.com/en-us/cpp/cpp/…
    – Robert
    Commented Feb 7 at 8:51
  • @Robert sorry, but that's only half the truth. Unicode is supposed to be a universal character set with encodings and previously Windows used UCS-2 (each code point encoded by exactly one 16-bit code unit) and now uses UTF-16 (usually Little Endian, due to supported machines) since XP. UTF-16 has limitations and IIRC is not capable of expressing every possible code point Unicode has to offer these days. As pointed out UTF-16 knows surrogates and is thus indeed variable-length for each code point. However, because the string only contains ASCII each code point is 16-bit only.
    – 0xC0000022L
    Commented Feb 7 at 20:54
  • Welcome to RE.SE, Katastic Voyage: despite some lacking details, Robert is right. This is UTF-16 if encountered on Windows XP or newer or UCS-2 on older Windows versions. The main issue with the comment is that wchar_t (= 16-bit type when targeting Windows, 32-bit on every other target platform I know) is merely the code unit. Code units are the machine-level "cells" into which the code points for Unicode-defined characters are encoded. In UTF-8 the code unit is usually char, in UTF-16 (on Windows) wchar_t on other systems UTF-32 wchar_t. Due to this, ...
    – 0xC0000022L
    Commented Feb 7 at 20:58
  • ... "escape routes" exist that enable us to encode code points (often referred to as "characters") that go beyond the value range of the underlying code unit to be encoded into multiple code units. These are then called surrogates. Reserved code points exist to "announce" these surrogates within a Unicode-encoded (whichever encoding!) string. UCS-2 (i.e. original Windows NT up to 2000) limited itself to the BMP which amounted to 1 wchar_t == 1 code point (for Windows, elsewhere wchar_t differs).
    – 0xC0000022L
    Commented Feb 7 at 21:03

1 Answer 1

2

This is most likely a "wide character", wchar_t[] type.

These are always 16 bits long and they are used to allow international characters and symbols that are at most 16 bit long, that is, the UTF-16 range.

The fixed length allows faster processing than multibyte unicode.

ASCII characters that could be encoded as 1 byte in multibyte unicode take up 2 bytes in widechar, which "wastes" an extra byte that is always null.

3
  • Wait, what? That makes no sense, several aspects. The wchar_t in Windows is the code unit to encode Unicode code points. And the full range of UTF-16 is very much supported. Yes, it has limitations compared to UTF-8 and UTF-32, but it is very much a Unicode encoding. UTF-16 has been used by Windows since XP (AFAIK) and previously UCS-2 was used, which can be considered sort of a predecessor to UTF-16. UCS-2 only ever considered the Basic Multilingual Plane (BMP) and thus limited itself to the notion of 1 wchar_t (16 bit) == 1 code point. Where the \0 is depends on Endianess.
    – 0xC0000022L
    Commented Feb 7 at 21:08
  • 1
    Thanks. I edited my answer. The null byte location depends on endianness, but the architectures Windows can run on are all little-endian.
    – Yotamz
    Commented Feb 7 at 21:54
  • these days, that is true. Agreed. But it wasn't always and UTF-16 (or Unicode in general) provides support for all of them. My main point was, however, that wchar_t is just the code unit and UTF-16 (since XP) does provide means to encode code points beyond a value range of 0..65535 through surrogates. So on modern Windows "characters" (code points) beyond the value 65535 are possible by encoding them via surrogate mechanism.
    – 0xC0000022L
    Commented Feb 8 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.