At the professional level, for what purpose is reverse software engineering used? What software is targeted and why?

For reasonably complex compiled code that's doing something novel, making meaningful insights into how that code operates via reverse engineering seems like it would be enormously intensive of expertise, labor, and time. In particular, I imagine that hiring competent assembly programmers is extremely difficult and possibly expensive. And yet, I haven't the foggiest idea where entities with the resources to do so would want to spend those resources.

This is my list of possibilities...

  1. Writing malware
  2. Writing counter-malware
  3. Maybe analyzing competitors products?

It's not a great list. What is the reality here? What sort of software justifies the expense to be reverse engineered?

See the comments on 0xC0000022L's answer for some refinement of the question.

up vote 39 down vote accepted

Reverse engineering is commonly used in many ways. Here is a list of just some of the most common professional activities where reverse engineering is involved.

  1. Malware research. Withouth doing reverse engineering, it's hard to determine what an actual piece of malware does, how and, more important, how to clean it or prevent infections.
  2. Plagiarism detection: Reverse engineering is often legally used to determine if company or particular X plagiarized code from company or particular Y.
  3. Forensics.
  4. Compatibility. Reverse engineering supported, and still does, many open source projects like Samba or Star/Libre/OpenOffice. Without professional (and hobbistic) reverse engineers, nobody would have been able to write a software like the previous ones as there was no specification available at all for the protocols or file formats involved. If you're using Libre/OpenOffice or MacOSX or Linux, you're actually using the result of the work of many reverse engineers during many years.
  5. Vulnerability research/development. In order to discover undocumented features, vulnerabilities or backdoors left by the company, programmers or a 3rd party, reverse engineering is naturally required.
  6. Modification of legacy programs. This is one of the most other common tasks I have seen in this field: company X bought program Y but the company producing it, Z, disappeared and they have no source code at all. That company X contracts reverse engineering services to fix bugs and/or adapt the old (dead) application for their own purposes. If the program was also protected, the reverse engineers would also need to first break the the copy protection.
  7. Analysing competitors products. Naturally, this is a very common thing. Specially when talking about hardware products. And it, by the way, happens in all fields. Do you think that Porsche doesn't reverse engineer Ferrari cars and the other way around?
  8. Source code recovery. Company X lost the source code of one of their programs/products for whatever reason and they only have the binaries. That company contracts reverse engineering services to reconstruct/recover the source code.
  9. Analysing foreign governments hardware and software. You get the idea.
  10. Binary patches development. Some products still evolve but the providing companies are very slow fixing bugs (Oracle comes to mind...) or simply don't care. In this case, reverse engineering services can be contracted to develop binary patches to be applied to the product in order to fix bugs or vulnerabilities (patches to be made again on each release).

I could continue writing down more professional task that reverse engineers often do, but I think this list gives you a general idea.

  • 3
    +1! The list is well thought out and the explanations are both informative and succinct. – Praxeolitic Oct 15 '14 at 8:28
  • 8
    Rumor (and webserver access logs indicating that it was used as training material inside Microsoft) has it that at one time (approximately 10-15 years ago), the only documentation (including inside Microsoft itself!) of the NTFS filesystem on-disk format was the Linux-NTFS Wiki, created through reverse-engineering. The same has been claimed for Samba. So, if that is in fact true, then Microsoft had to reverse engineer their own protocols in order to comply with the EU antitrust ruling, forcing them to publicly document their network protocols. – Jörg W Mittag Oct 15 '14 at 9:16
  • @JörgWMittag, if that's true, I wonder how MS was planning on adding to NTFS? Were they planning on reverse-engineering it themselves or just adding to the filesystem without knowing what they were adding to? Knowing MS, I assume it's the latter. – trysis Oct 15 '14 at 15:24
  • 4
    @trysis: They probably had a team of Devs that knew it inside and out, and it simply never occurred to anyone that no documentation had been written. We discovered that at our work recently when an employee left and then we realized nobody knew anything about the code he had maintained. – Mooing Duck Oct 16 '14 at 18:38
  • 3
    @MooingDuck you may be surprised how often it happens. Event to the biggest software companies. – joxeankoret Oct 17 '14 at 7:38
  1. Finding undocumented features of a product.

    • for instance for locating a registry key or setting to turn on a debug mode.
  2. Finding how exactly a poorly documented feature works.

  3. Debugging

    • sometimes i find myself loading my own software in IDA to find what code the compiler produced, either to find bugs, or to see how an optimization worked out.
  4. Cracking copy protection

    • keygens are usually created by reverse engineering an application's licensekey validation algorithm.
    • sometimes a company may have lost licene codes to some ancient piece of software.
  5. Evaluating the security of a closed source product.

  6. Modify an existing product

    • like flashing your own custom rom to a phone.
  7. Interface current products with ancient software.

  • 4
    I can confirm number 6. Done that many times, even though the source exists. Simple reason: optimizers sometimes screw up. For the same reason I have used GDB in disassembler mode instead of source mode. – 0xC0000022L Oct 15 '14 at 7:14
  • Really? Are all of these professional activities? #4 and #7 especially raise an eyebrow. – Praxeolitic Oct 15 '14 at 7:16
  • 2
    @Praxeolitic: absolutely (4). Take Windows for example. How do you get to know the innards in absence of the source code? And 7 has even a completely legit use case: have someone try your copy protection scheme to see whether it withstands anything. Heck lock manufacturers do that. – 0xC0000022L Oct 15 '14 at 7:40
  • ok, #7 is questionable ... but adding #10 – Willem Hengeveld Oct 15 '14 at 8:56
  • 2
    Re. #4: I've encountered libraries that were so badly documented that I had to resort to RE to figure out how to use the features that were meant to be public. You see this kind of thing a lot when dealing with hardware companies that don't know how to do software. – Kevin Krumwiede Oct 15 '14 at 18:58

As a professional Reverse Engineer I have reveresed for:

  • Troubleshooting
  • Exploit Development
  • Malware Analysis
  • Implementing badly documented API calls.
  • Plagiarism , they knew of an undocumented Kernel call within the Nokia phonebase. Thus we reverse engineerd this and created a program that used this call as well.

Overall reversing increases your knowledge in programming.

A key purpose in reverse engineering it to recover an abstract design of the artifact.

A source program is an implementation of a concept. It contains a huge amount of code/gates that achieve an effect, and then optimizations on top of that to achieve the effect reasonably well (space, time, heat, area, power, ...). It is focused on good execution, not good documentation of the concepts.

That's all well and fine for the running product. But if you want understand the code you have (and you have made the mistake of throwing the design away [everybody has, don't get me started on the stupidity of this]), then abstracting away all that detailed implementation, leaving behind the abstract structure and computation, is extremely helpful.

Here's technical paper on how to do this in a regular way, that I wrote on the topic (last century [!], still good IMHO): Reverse Engineering is Reverse Forward Engineering. This paper takes low level assembly code from a real time OS, and abstracts it back into the synchronization primitive it implements. In essence what it says is, "This concept was realized in the code like this at these places in the code".

This idea is partially derived from Rich and Water's famous(?) mid 1980's research on The Programmer's Apprentice. In their case they were focused on finding realizations of abstractions in code, by doing what amounts to data flow graph matching. (There have been a variety of other papers and Ph.D. theses suggesting similar ideas). My RE paper suggests grafting an entire design model on top of low level recognition.

My company recently has recently implemented Rich&Water's vision, for Dow Chemical, extracting process control models from low level industrial controller code. Dow is using this to migrate legacy code that run chemical plants, to modern controllers. See SD building tools for Dow Chemical. This only announces the work, but we've been at it for a few years and are getting good results. We hope to be writing this up for technical journals soon, as well as applying this to more conventional languages such as C.

I assert that all these ideas apply to hardware and codesign too.

  1. Writing cheats for games.
  2. Writing an anti-cheat for games.
  3. Looking for badly written code to exploit or to fix and patch it.
  1. Honing your skills and keeping up-to-date with them (as well as with using the tools).
  2. Curiosity. Yep, doesn't have to be purely goal-driven like 4. and 5. in Willem's answer.
  3. Establishing compatibility with a legacy product, file format or architecture. Motivation can be 11 in that case or 4 or 5 or possibly something completely different.
  • But these are non-professional reasons, right? What I'm really having trouble understanding is how programs like IDA, that are out of the price range of an individual, exist. What is the incentive? I of course have tons of sympathy for 10 & 11, but I want to try to make this a question with a well defined answer: what are the primary incentives for companies/governments to engage in reverse engineering? – Praxeolitic Oct 15 '14 at 7:22
  • @Praxeolitic: you might not let 11 slip through, although that shows how much opinion-based your question really is, but 10 is certainly a professional reason. Unless you think that a professional doesn't need to keep up-to-date and train his/her skills. Side-note: even as a student I bought IDA and later I kept the license updated. For over a decade now. And yes, as an individual. Curiosity can be a strong drive. Please define professional. Do you mean commercial? – 0xC0000022L Oct 15 '14 at 7:25
  • The question is of course subjective question to an extent but I'm trying to be polite and not just ask for a laundry list with my first question here. Oy, it's a young stackexchange, maybe this isn't so bad. – Praxeolitic Oct 15 '14 at 7:32
  • Don't worry, I was tempted to vote for closing it, but I am also very excited to see others come up with more points. Because I think you and Willem basically nailed it, with variations possible of course (which is why I think 11 is more of a superset, whereas the motivation can be commercial or not). Nevertheless the reasons are but a short list. – 0xC0000022L Oct 15 '14 at 7:37
  • 1
    Reverse Code Engineering, which limits the scope to programs or "code" in general, as opposed to reverse engineering which could be taking a laundry machine apart to understand how it's built and how it does what it does. This SE site is about the latter, not just the former. That's why this matters :) – 0xC0000022L Oct 15 '14 at 7:51

I have carried out reverse engineering projects professionally a fair bit. Generally we are commissioned to:

1) Determine how hard it would be to create a crack for some piece of software (by the developers of that software).

2) Understand a specific exploit (in the security sense, e.g. a buffer overflow). In other words, "here's the email we got; tell us what happened", or "here's the exploit. How, exactly, does it work and how can we defend against this kind of exploit in future?"

3) Find bugs in a piece of software by examining the binary without reference to the source code, as part of a larger security audit of the code. This is surprisingly useful because some kinds of dangerous behaviour that aren't obvious in code can "leap out" at you in the binary, e.g. complex preprocessor macros causing dangerous behaviour in C/C++. So, counterintuitively, we sometimes use reversing because it's faster. Check out IDA pro and Hopper for the sorts of "quick reversing" features that help in this area - decompiling to pseudocode, finding all calls to X or just searching the string table then finding all references to a dangerous/suspect looking string, and so on. Sometimes just running 'strings' on a binary can give you ideas as to where the bugs might be; following that up with a real, pro reversing tool can help. This isn't restricted to C/C++/Objective C - if we're working in .Net or Java, we'd use .Net Reflector or the Java Decompiler and other environments have their equivalents.

Sometimes we use reverse engineering to help us in a larger project. For instance, we might want to test a component in isolation; to do this without the source code we might want to edit the binary to introduce logging calls or shortcut some behaviour.

Then there's fuzzing, of course, but that's probably not really reversing in the true sense, although it can involve examining code at the assembly level, to understand why the code crashed / where the bug is / whether it's a security problem.

There are many, many reasons why reverse engineering is helpful even in a general development context. In the security field, it's pretty much essential.

In terms of professional development, if you're in the security field and you have good reversing skills, you will probably be at the upper end of the pay scale. You need to be able to do the other stuff, but if you have reversing on your CV - and you can back it up - then getting hired shouldn't be a problem (in the security field at least).

There are already a lot of great answers. If I may add my two cents. Reverse engineering software is akin to the mechanic or tinkerer whom just enjoys taking things apart, understanding how they work, putting them back together, and possibly modifying their subject to adapt its behavior so it is more to their liking. There is no shortage of examples in the physical realm.

Any Electrical Engineer, hardware designer, and the most eliete QA engineers are talented reverse engineers; which is to say, they have an expertise in debugging/analyzing software/hardware implementations. I interned at AMD as I worked my way through school and my debugging skills not only helped me advance to more serious roles within the company, but it is also really fun! Like a game or puzzle that you can play for as long as you want. The challenges will never stop coming.

My favorite example of a legitimate need for reverse engineers is NASA. They are not the only outfit where the phrase "mission critical" applies, where people's lives depend on the quality of their work, but it is the only one that has to fix problems even if their product is hurtling through space. They have to set and freeze product versions years before that software / hardware ever goes to the launch pad. That may seem silly, because the phones in our pocket have more computing power than the entire shuttle, but they can't use modern technology. It's too risky. They have to accept what is available at the time that the specs are frozen and that's it - lest they incur the scheduling and workload that upgrading, testing, fixing, re-testing, verifying again, simulating realistic conditions to determine failure points, fixing it, and starting over.... it is serious business and those engineers do not take their responsibility lightly.

Would you be surprised to learn that NASA projects have used Microsoft OSes on hardware went into shuttle components? It is not much different than using a M$ OS as the foundation for, say for instance, a point-of-sale terminal or for the control system at a power plant. Now consider a scenario where the OS, the hardware, or some portion of third-party software is found to have a flaw in it. Moreover, imagine that the vendor does not care about the flaw to the degree that NASA does and decides that there is no fiscal motivation for them to fix the issue - even if lives depend upon that component not failing. NASA is stuck.

At this point their options are to either replace the component and start their processes all over - a huge cost in time, man-hours, and no guarentee that they won't discover another critical flaw after upgrading; or, instead of upgrading, they can reverse engineer the hardware/software failure and determine if they could fix the problem themselves - maybe it would take some curcuit re-design, changing the values of a few resistors so that voltage levels on the memory bus stayed within required limits under the extreme conditions that were making them fail or maybe there is a binary patch that they can apply that to mitigate the error state they discovered where life-support systems began failing or a probe's batteries drained and left it unable to re-charge itself.

I'm not totally making these scenarios up. They come from stories I have been told, by friends who worked at JPL, my engineering professors when I was in the University, or colleagues that have worked with in the tech industry. All this, and more, has been surmounted by NASA's engineers. It did cost quite a bit more than India paid (tip of the hat to them too), to do what they have done, but the first time through is always more costly.

Technically, NASA engineers would be violating one or more of the intellectual property laws that vendors claim they have (they may or may not actually have those rights; I"m not trying to debate that here). If you bought a corvette, you would be completely within your rights to bore out the cylinders to change your car's performance profile. Different tires. Modified suspension. Tweaked timing and mixture ratios in the fuel injection system. The famous Shelby Cobra is not only modified, but also remarketed with its aftermarket modifications- all entirely legal.

If you bought a radio and wanted to make some modifications, so that it was water proof and you could take it on the lake with you. Amplify it's output so that it would still be audible over the boat engine, or anything of that sort... legal.

Now, jailbreak your iPhone, modify your xBox or the Kinect system that came with it, or, in NASA's case, modify the OS and/or re-engineer the memory bus on a motherboard, to ensure the safety of a human mind you. All of these activities have evoked fierce reactions from the vendors that, for whatever their reasoning, do not want those activities to be continue nor become more widespread. Ultimately, this led to the DMCA laws that we have today being passed. Before that, even if you were clicking "I agree" to the rediculus terms and services that were put before you there was still the question if that contract was legal enforceable.

The DMCA, which does nothing to disuade the massive blackmarkets in China or South America, carries harsh penalties and has been treated as a kind of "hunting license" for those that wish to go after consumers can and have. Should it be illegal to make unlicensed copies of someone elses music; to sell, trade, or even give it away?

Clearly, that should be illegal. Artwork, books, patented processes, protected trademarks; yes, those should be protected too. Software and hardware; again, yes the the author of software should be able to enforce resonalbe aspects of their licensing that protect the the investment that have made to create said software or hardware. Explicitly, it is, and should remain, illegal to puchase one copy legitimately, turn around and make copies, and then sell/trade/or give the unlicensed copies. Microsoft, VMWare, Adobe, and similar big companies should retain that right.

Now, once I have licensed the software or hardware can I make modifications and then sell/trade/or give away the software/hardware that featured my modifications. That depends. It i, and should be illegal, to profit from derivatives works beyond the limit to which you originally licended the art (i.e. software). For example, you could not draw the likeness of a Disney character, change the name, and then claim it as your own and profit from it's use. I believe that Mikey no longer protected, but when he was that was and should be illegal.

Should a computer hobbyist, like the ones that we credit for founding the industry today, be allowed to to explore, tinker, change, discuss, show, or teach any manner of activity that they desire on their legally licensed software and hardware? I won't answer that, but I will say that the DMCA says "no". In fact, the same tools that a reverse engineer would use for debugging the operating system are the very same tools that the prosecution will cite as evidence of illegal activity.

Let's be a little more giving for someone prosecuting a DMCA case where the accused was know to have accessed the cryptographic keys that, among other things, would allow the defendant to make copies of protected art (e.g. music and movies). There are some legitimate reasons that could explain the defendants actions, but not many and the illegal activities that could be pursued are costly to those that are trying to protect themselves. At the end of the day, even though they tried their best, there is no way to actually prevent anyone from accessing those cryptographic keys.

It reminds me of the tag on a mattress waring of insanely harsh penalties for removing said tag, but there is no protection for the tag. Just threats and overreaching penalties that are used to make examples out of as they stoke the fire and declare a witch hunt.

Should Apple be able to prosicute a security research whom respectfully, and sincerely trying to help and support Apple make better products, be prosicuted by the technology giant? Hell no, but you better believe that they tried - that happened. The employees of Target or Home Depot that warned of potential dangers; where are they now? They were fired, run-off, considered a nuisance to the company, shunned, and labeled as "not a team player". In the wake of the data breaches that Target, Home Depot, and countless others (certainly at least one of these events has impacted your life) it is more apparent now than ever that reverse engineering is not only a right, but an aspect of our emerging social structure that is going to be needed more than ever to 1) protect the innocent consumers and digital citizens of the world like yourself and 2) hold accountable those whom are responsible to take reasonable measure to protect the community that entrusted them with their well being (granted in exchange for a service).

The NSAs, the CIAs, the DeutcheBanks, the HealthCare.govs, and the hospitals that we all wanted and empowered to do what they have done. It is the reverse engineer who will become the modern freedom fighter, tomorrows activist, that is going to be on your side, the little guy, when power corrupts and greed lets way to disregard for the responsibilities that were implicit in the power granted to those organizations. To be just a bit more dramatic about it; reverse engineers are the priests of a new religion. We should honor them, applaud their bravery standing up to those drunk on power, and protect them when they risk the privacy of their family to step into the line of fire, on our behalf, because it is "the right thing to do".

If you know one of these individuals- buy them a beer tonight and say thank you. As much as any soldier, they are going to be the ones that protect us when the next digital tragedy is impending.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.