Skip to main content
Reverse Engineering

Return to Answer

typos corrected
Source Link
edit approved Nov 18, 2018 at 22:12
Glorfindel
edit approved Nov 18, 2018 at 22:12
Glorfindel
  • 232
  • 1
  • 2
  • 10

There is no similar feature for javaJava byte code.

In javaJava bytecode, there is no neccesitynecessity to do this. Function names, variable names, and similar information is present in the compiled bytecode anyway. The "standard library", rt.jar, isn't embedded into the bytecode either, so if a class uses an ArrayList, the class will have a reference to java.util.ArrayList even after the obfuscation process. So, nothing to do here for a signature analyzer.

If the application choseschooses to obfuscate the standard library as well, it needs to include that obfuscated library into its own jar files. This will probably raise some licensing issues, but apart from that, as the obfuscator will rename methods and fields, the byte code of that obfuscated library will be too different from the original byte code to be recognizable by a feature like IDA's FLIRT. Also, you can't just rename the method in IDA, because you'd have to modify all the references as well.

However, there is at least one open source project that has a similar problem like yours, and they seem to have solved it quite well. Minecraft is a popular game that includes a server that's written in javaJava (and obfuscated); the Spigot project decompiles this server, changes some things, adds an API, and distributes the result. Specifically, to avoid licensing issues, they distribute a build system which downloads, decompiles, patches, and recompiles the minecraftMinecraft server on the user's machine.

The source code, which is generated using these maps, is obviously way more readable than the original one. And, at least for the spigot project, recompiling this source (after adding some patches to create an API) yields a working minecraftMinecraft server.

So, maybe, this could be a way for you to proceed - use fernflower to decompile your classes, load them into an editor to find some useful code and assign readable class names, write a mapping file, and repeat this a few times. Then, when you want to do some dynamic analysis, recompile the decompiled javaJava code and load that into IDA.

  • you will still have to identify each function manually - but as i said, you probably won't find any standard library functions in your code anyway. And, there are no existing signature libraries for whichever method you use, so there's probably no way around that.
  • the decompiled/recompiled code may not work, because of bugs/shortcomings in the javaJava decompiler itself
  • the decompiled/recompiled code may not work because of the obfuscator; for example, the obfuscator might replace all string constants with something like Obfuscator::decode("some_crypted_stuff"), where the decode function uses the name of the calling class as its decryption key, meaning the decryption fails when the class gets renamed
  • the obfuscator might bring its own class loader, which mangles the class name before loading it; for example, it might know to turn a com.obfuscate.SOME_BASE_64_STRING class into the decoded base64 string, so com.obfuscate.amF2YS51dGlsLkFycmF5TGlzdAo.something() would call java.util.ArrayList.something(). This breaks the visible connection between caller and callee (but the name mapping could solve this problem quite well and you could automate a lot)

It's questionable if these approaches beat manual reverseengineeringreverse engineering within IDA, but as IDA doesn't have this feature, and as there are no generic signature libraries (and they wouldn't work anyway, see above), it's the best i have to offer.

Turns out iI have a little project where iI could use some deobfuscation mapping myself, and checked into the deobfuscation process of the minecraftMinecraft decompiling step a bit more.

There is no similar feature for java byte code.

In java bytecode, there is no neccesity to do this. Function names, variable names, and similar information is present in the compiled bytecode anyway. The "standard library", rt.jar, isn't embedded into the bytecode either, so if a class uses an ArrayList, the class will have a reference to java.util.ArrayList even after the obfuscation process. So, nothing to do here for a signature analyzer.

If the application choses to obfuscate the standard library as well, it needs to include that obfuscated library into its own jar files. This will probably raise some licensing issues, but apart from that, as the obfuscator will rename methods and fields, the byte code of that obfuscated library will be too different from the original byte code to be recognizable by a feature like IDA's FLIRT. Also, you can't just rename the method in IDA, because you'd have to modify all the references as well.

However, there is at least one open source project that has a similar problem like yours, and they seem to have solved it quite well. Minecraft is a popular game that includes a server that's written in java (and obfuscated); the Spigot project decompiles this server, changes some things, adds an API, and distributes the result. Specifically, to avoid licensing issues, they distribute a build system which downloads, decompiles, patches, and recompiles the minecraft server on the user's machine.

The source code, which is generated using these maps, is obviously way more readable than the original one. And, at least for the spigot project, recompiling this source (after adding some patches to create an API) yields a working minecraft server.

So, maybe, this could be a way for you to proceed - use fernflower to decompile your classes, load them into an editor to find some useful code and assign readable class names, write a mapping file, and repeat this a few times. Then, when you want to do some dynamic analysis, recompile the decompiled java code and load that into IDA.

  • you will still have to identify each function manually - but as i said, you probably won't find any standard library functions in your code anyway. And, there are no existing signature libraries for whichever method you use, so there's probably no way around that.
  • the decompiled/recompiled code may not work, because of bugs/shortcomings in the java decompiler itself
  • the decompiled/recompiled code may not work because of the obfuscator; for example, the obfuscator might replace all string constants with something like Obfuscator::decode("some_crypted_stuff"), where the decode function uses the name of the calling class as its decryption key, meaning the decryption fails when the class gets renamed
  • the obfuscator might bring its own class loader, which mangles the class name before loading it; for example, it might know to turn a com.obfuscate.SOME_BASE_64_STRING class into the decoded base64 string, so com.obfuscate.amF2YS51dGlsLkFycmF5TGlzdAo.something() would call java.util.ArrayList.something(). This breaks the visible connection between caller and callee (but the name mapping could solve this problem quite well and you could automate a lot)

It's questionable if these approaches beat manual reverseengineering within IDA, but as IDA doesn't have this feature, and as there are no generic signature libraries (and they wouldn't work anyway, see above), it's the best i have to offer.

Turns out i have a little project where i could use some deobfuscation mapping myself, and checked into the deobfuscation process of the minecraft decompiling step a bit more.

There is no similar feature for Java byte code.

In Java bytecode, there is no necessity to do this. Function names, variable names, and similar information is present in the compiled bytecode anyway. The "standard library", rt.jar, isn't embedded into the bytecode either, so if a class uses an ArrayList, the class will have a reference to java.util.ArrayList even after the obfuscation process. So, nothing to do here for a signature analyzer.

If the application chooses to obfuscate the standard library as well, it needs to include that obfuscated library into its own jar files. This will probably raise some licensing issues, but apart from that, as the obfuscator will rename methods and fields, the byte code of that obfuscated library will be too different from the original byte code to be recognizable by a feature like IDA's FLIRT. Also, you can't just rename the method in IDA, because you'd have to modify all the references as well.

However, there is at least one open source project that has a similar problem like yours, and they seem to have solved it quite well. Minecraft is a popular game that includes a server that's written in Java (and obfuscated); the Spigot project decompiles this server, changes some things, adds an API, and distributes the result. Specifically, to avoid licensing issues, they distribute a build system which downloads, decompiles, patches, and recompiles the Minecraft server on the user's machine.

The source code, which is generated using these maps, is obviously way more readable than the original one. And, at least for the spigot project, recompiling this source (after adding some patches to create an API) yields a working Minecraft server.

So, maybe, this could be a way for you to proceed - use fernflower to decompile your classes, load them into an editor to find some useful code and assign readable class names, write a mapping file, and repeat this a few times. Then, when you want to do some dynamic analysis, recompile the decompiled Java code and load that into IDA.

  • you will still have to identify each function manually - but as i said, you probably won't find any standard library functions in your code anyway. And, there are no existing signature libraries for whichever method you use, so there's probably no way around that.
  • the decompiled/recompiled code may not work, because of bugs/shortcomings in the Java decompiler itself
  • the decompiled/recompiled code may not work because of the obfuscator; for example, the obfuscator might replace all string constants with something like Obfuscator::decode("some_crypted_stuff"), where the decode function uses the name of the calling class as its decryption key, meaning the decryption fails when the class gets renamed
  • the obfuscator might bring its own class loader, which mangles the class name before loading it; for example, it might know to turn a com.obfuscate.SOME_BASE_64_STRING class into the decoded base64 string, so com.obfuscate.amF2YS51dGlsLkFycmF5TGlzdAo.something() would call java.util.ArrayList.something(). This breaks the visible connection between caller and callee (but the name mapping could solve this problem quite well and you could automate a lot)

It's questionable if these approaches beat manual reverse engineering within IDA, but as IDA doesn't have this feature, and as there are no generic signature libraries (and they wouldn't work anyway, see above), it's the best i have to offer.

Turns out I have a little project where I could use some deobfuscation mapping myself, and checked into the deobfuscation process of the Minecraft decompiling step a bit more.

added 1680 characters in body
Source Link
Guntram Blohm
Guntram Blohm
  • 13k
  • 2
  • 22
  • 33

Edit 2016-04-17

Turns out i have a little project where i could use some deobfuscation mapping myself, and checked into the deobfuscation process of the minecraft decompiling step a bit more.

Turns out they don't even use fernflowers remapping feature. Instead, they have their own set of tools. These are lamentably underdocumented, but there seems to be a GitHub Repository.

The first one, SpecialSource.jar, compares an obfuscated jar with an unobfuscated version, and generates a mapping table. This seems to allow to build new tables quickly when a new version of the obfuscated original comes out. There is, however, no documentation about what's exactly being compared.

The second one, SpecialSource-2.jar, remaps class and method references and names directly in the jar file, without a decompilation/compilation step, so it avoids all the problems that come with that. So once you have a mapping file, you can apply it to the jar to get an output jar, which you can then feed into IDA, or a decompiler of your choice. Usage is

java -jar /path/to/SpecialSource-2.jar map -m mapping.csrg \
-i obfuscated.jar -o readable.jar

with the mappings file looking like this

# This renames a class, given the obfuscated name and the readable one
com/example/ab/a MyNiceClassName
# This renames a method, given the class, method, signature, and readable name.
MyNiceClassName a (D)I doubleToInt

Edit 2016-04-17

Turns out i have a little project where i could use some deobfuscation mapping myself, and checked into the deobfuscation process of the minecraft decompiling step a bit more.

Turns out they don't even use fernflowers remapping feature. Instead, they have their own set of tools. These are lamentably underdocumented, but there seems to be a GitHub Repository.

The first one, SpecialSource.jar, compares an obfuscated jar with an unobfuscated version, and generates a mapping table. This seems to allow to build new tables quickly when a new version of the obfuscated original comes out. There is, however, no documentation about what's exactly being compared.

The second one, SpecialSource-2.jar, remaps class and method references and names directly in the jar file, without a decompilation/compilation step, so it avoids all the problems that come with that. So once you have a mapping file, you can apply it to the jar to get an output jar, which you can then feed into IDA, or a decompiler of your choice. Usage is

java -jar /path/to/SpecialSource-2.jar map -m mapping.csrg \
-i obfuscated.jar -o readable.jar

with the mappings file looking like this

# This renames a class, given the obfuscated name and the readable one
com/example/ab/a MyNiceClassName
# This renames a method, given the class, method, signature, and readable name.
MyNiceClassName a (D)I doubleToInt
Bounty Ended with 100 reputation awarded by Dr Deo
Source Link
Guntram Blohm
Guntram Blohm
  • 13k
  • 2
  • 22
  • 33

There is no similar feature for java byte code.

When you compile a C program, and statically link it to a standard library, the library code will be present, more or less unmodified, within the binary (except for addresses which will change), but there won't be any hint that a particular function had a particular name before being compiled (unless debugging is enabled while compiling/linking). But, any given function has a more or less fixed pattern of bytes, and by recognizing these bytes, IDA can assign the original name to the function.

In java bytecode, there is no neccesity to do this. Function names, variable names, and similar information is present in the compiled bytecode anyway. The "standard library", rt.jar, isn't embedded into the bytecode either, so if a class uses an ArrayList, the class will have a reference to java.util.ArrayList even after the obfuscation process. So, nothing to do here for a signature analyzer.

If the application choses to obfuscate the standard library as well, it needs to include that obfuscated library into its own jar files. This will probably raise some licensing issues, but apart from that, as the obfuscator will rename methods and fields, the byte code of that obfuscated library will be too different from the original byte code to be recognizable by a feature like IDA's FLIRT. Also, you can't just rename the method in IDA, because you'd have to modify all the references as well.

However, there is at least one open source project that has a similar problem like yours, and they seem to have solved it quite well. Minecraft is a popular game that includes a server that's written in java (and obfuscated); the Spigot project decompiles this server, changes some things, adds an API, and distributes the result. Specifically, to avoid licensing issues, they distribute a build system which downloads, decompiles, patches, and recompiles the minecraft server on the user's machine.

They use the fernflower decompiler, which has an option to rename symbols while decompiling, and include an extensive map of obfuscated-to-readable symbol names. Example:

AttributeInstance a ()LIAttribute; getAttribute
AttributeInstance a (D)V setValue
AttributeInstance e ()D getValue
AttributeMapServer b ()Ljava/util/Set; getAttributes
AxisAlignedBB b (DDD)LAxisAlignedBB; grow

The source code, which is generated using these maps, is obviously way more readable than the original one. And, at least for the spigot project, recompiling this source (after adding some patches to create an API) yields a working minecraft server.

So, maybe, this could be a way for you to proceed - use fernflower to decompile your classes, load them into an editor to find some useful code and assign readable class names, write a mapping file, and repeat this a few times. Then, when you want to do some dynamic analysis, recompile the decompiled java code and load that into IDA.

Of course, this approach still has some problems:

  • you will still have to identify each function manually - but as i said, you probably won't find any standard library functions in your code anyway. And, there are no existing signature libraries for whichever method you use, so there's probably no way around that.
  • the decompiled/recompiled code may not work, because of bugs/shortcomings in the java decompiler itself
  • the decompiled/recompiled code may not work because of the obfuscator; for example, the obfuscator might replace all string constants with something like Obfuscator::decode("some_crypted_stuff"), where the decode function uses the name of the calling class as its decryption key, meaning the decryption fails when the class gets renamed
  • the obfuscator might bring its own class loader, which mangles the class name before loading it; for example, it might know to turn a com.obfuscate.SOME_BASE_64_STRING class into the decoded base64 string, so com.obfuscate.amF2YS51dGlsLkFycmF5TGlzdAo.something() would call java.util.ArrayList.something(). This breaks the visible connection between caller and callee (but the name mapping could solve this problem quite well and you could automate a lot)

Even if you can't get a decompiled, recompilable, version of your binary, you can probably write some code that works on the decompiled source to identify functions by their behaviour, using some heuristics, Then, have your code write a mapping table, and write a python plugin to import that mapping table into your code as comments.

It's questionable if these approaches beat manual reverseengineering within IDA, but as IDA doesn't have this feature, and as there are no generic signature libraries (and they wouldn't work anyway, see above), it's the best i have to offer.