Most of file formats IDA handles are structured in a linear way, for example EXE or ELF or Java's Class file and so on.
However when it comes to tree-structured file format, I can't come up with a good way to properly handle them.
For example, recently I met a obfuscated PYC file, and the code has an OLLVM-like obfuscation plus some junk bytes, so a strong disassembler like IDA is needed. The PYC file format is in fact some magic byte + python marshal data, which is easy to parse. However, the marshaled data is in tree structure. Here's an example: http://www.beesfun.com/2017/04/01/PyCodeObject%E5%92%8Copcode/
As you can see in the parsed result, the root object is a module whose consts contain other functions in module, and the function in module can also have sub-functions in their consts.
Also dis module or pytype can be used to parse the PYC file
Like Java's compiled file, PYC stores bytecodes and some metadata for each functions, functions call each other with name instead of address, bytecode believe the address starts from 0 in every function. So I studied the source code of module/java in idasdk73, trying to understand how the developers from HexRays cope with the Java class file format.
Then I found Java module in IDA are making a separate segment for each method and slot area. Because the python bytecode assume address starts from 0 in every function, and in every function there's a separate const pool which is also indexed from 0, I decided to create a header segment, a code segment and a const segment.
But this same way won't work for a PYC file, as the PYC format is tree-structured (sub-functions are stored in parent const pools), and IDA does not allow overlapped segments. So I can't make the consts area a single segment.
But if I create a segment for each const, there will be tens of thousands of segments if the original py file is big.
Is there a better way to make a loader for the tree-structured format like PYC?
