IDAPython Strings constantly returns NoneType with str()

Question

I'm trying to automate disassembly of a firmware image using IDA Pro 6.5 and IDA Python. One of the process I want to implement is to locate strings and create a data segment around them.

Using the GUI, I have little issue doing so. However when using the idautils.Strings() API call, I can retrieve a list of StringItem objects, but I fail to access the actual string data with str() or unicode(). Below is the failing function, which is taken from IDA Python Google Code archive:

def find_strings():
    s = idautils.Strings(False)
    s.setup(strtypes=Strings.STR_UNICODE | Strings.STR_C)
    for i, v in enumerate(s):
        if v is None:
            print("Failed to retrieve string index %d" % i)
        else:
            print("%x: len=%d type=%d index=%d-> '%s'" % (v.ea, v.length, v.type, i, str(v)))

When ran into IDA, the following error is reported:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 8, in find_strings
TypeError: 'StringItem' object is not callable

When replacing the str(v) argument with the constant aaa in the print function, I get a list of StringItem objects without any problem:

Python>find_strings()
208e: len=8 type=3 index=0-> 'aaa'
21b0: len=55 type=0 index=1-> 'aaa'
229d: len=6 type=0 index=2-> 'aaa'
22c5: len=5 type=0 index=3-> 'aaa'
22d3: len=33 type=0 index=4-> 'aaa'
...

If I attempt to use the unicode() function instead, I get the following error:

Python>find_strings()
208e: len=8 type=3 index=0-> '
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 8, in find_strings
TypeError: coercing to Unicode: need string or buffer, NoneType found

From my understanding, it seems that the StringItem contains no strings for an unknown reason (or an issue with the plugin, specific version of Python maybe?), however they are displayed in the GUI.

I'm seeking advice on either what I'm doing wrong, or an alternative way to extract the strings using the IDApython plugin. Thanks

Updates

The code above appears valid after adding the missing parenthesis as mentioned in the comments. However this was only a typo in the post and not the source of the issue. The find_strings worked fine in other typical binaries. Further proof is that by using the idc.GetString(self.ea, self.length, self.type) also returned NoneType.

Diff mentioned that the get_ascii_contents2 is failing and thus returning null, which is very likely the cause. What is unclear is why the function is failing, while the GUI succeeds in locating most of the strings.

The first string at 0x208E is a trash Unicode string. The string at 0x21B0 is an actual ASCII string composed of 37 chars. I cannot post the complete string due to disclosure/legal issues. Notice that when displayed in the hex editor, the byte order of the ASCII view is inverted for an unknown reason. The bitness of the overall firmware is 16bit.

434F 5059 5249 4748 5420 A920 ... 4544 2000 0000 : OCYPIRHG T ¬ ... DE.

Finally, note that the function MakeStr works without any issue. I have the following code, when used at 0x21B0, will successfully create a string within a data segment:

def create_string(self, _startea, _endea, _segname=".const", _unicode=False):
        
        if (SegStart(_startea) == idc.BADADDR):
            self.create_data_segment(_startea, _endea, ".const")
        else:
            segtype = GetSegmentAttr(_startea, SEGATTR_TYPE)
            if (segtype != IDAEngine.SEG_TYPE_DATA):
                DelSeg(_startea, 0)
                self.create_data_segment(_startea, _endea, _segname)
        
        result = MakeStr(_startea, _endea)
        if (result == IDAEngine.FAIL):
            print "[-] Failed to create a string at 0x{:x} to 0x{:x}.".format(_startea, _endea)

At this point, I believe the structure of the firmware is to blame (combination of bitness, lack of symbols and an obsolete but supported microprocessor), however I couldn't pinpoint the exact issue. For now, since I can use find_strings() to retrieve the offsets and then use MakeStr on strings with a certain length and the manually vetting the "real" strings.

Final Remarks

For posterity, I never really solved the issue, however I can confirm the underlying binary file was responsible for raising an exception in get_ascii_contents2. I've reloaded the same file, however as a raw binary file in one large segment and the function worked flawlessly.

Answer 1

This took some digging, however it appears you're hitting an interesting edge case that the original author of the scripts didn't consider.

str(StringItem) calls the following code inside idautils.py;

    def __str__(self):
        return self._toseq(False)

Which leads to _toseq in idautils.py;

    def _toseq(self, as_unicode):
        if self.is_2_bytes_encoding():
            conv = idaapi.ACFOPT_UTF16
            pyenc = "utf-16"
        elif self.is_4_bytes_encoding():
            conv = idaapi.ACFOPT_UTF8
            pyenc = "utf-8"
        else:
            conv = idaapi.ACFOPT_ASCII
            pyenc = 'ascii'
        strbytes = idaapi.get_ascii_contents2(self.ea, self.length, self.type, conv)
        return unicode(strbytes, pyenc, 'replace') if as_unicode else strbytes

If we dig into the get_ascii_contents2 inside py_bytes.hpp method we see that this method could actually return a NoneType if get_ascii_contents2 fails;

if ( !get_ascii_contents2(ea, len, type, buf, len+1, &used_size, flags) )
{
  qfree(buf);
  Py_RETURN_NONE;
}

Essentially, the code is fine, however you should add a check or exception handling if a str(StringItem) returns the with a TypeNone since it is possible for this type of value to be returned.

You could help debug further by providing what the hex data is at ea of 0x208e with the length of 8 as shown in your output;

208e: len=8 type=3 index=0->