Reverse Engineering an Encryption Algorithm

Question

I have a program that has the ability to encrypt files and I want to be able to create a program that can read these files. I believe it uses a Blowfish encryption, but I'm not certain on that. How can I find the type of encryption used and its key?

Using Reflector, I was able to find the following code:

public static unsafe string Encrypt(string text)
{
    string str;
    basic_string<char,std::char_traits<char>,std::allocator<char> > local2;
    sbyte modopt(IsSignUnspecifiedByte)* numPtr = (sbyte modopt(IsSignUnspecifiedByte)*) Marshal.StringToHGlobalAnsi(text);
    std.basic_string<char,std::char_traits<char>,std::allocator<char> >.{ctor}(&local2, numPtr);
    try
    {
        basic_string<char,std::char_traits<char>,std::allocator<char> > local;
        IntPtr hglobal = new IntPtr(numPtr);
        Marshal.FreeHGlobal(hglobal);
        basic_string<char,std::char_traits<char>,std::allocator<char> >* localPtr = _Encrypt(&local, (basic_string<char,std::char_traits<char>,std::allocator<char> > modopt(IsConst)* modopt(IsImplicitlyDereferenced)) &local2);
        try
        {
            str = new string(std.basic_string<char,std::char_traits<char>,std::allocator<char> >.c_str((basic_string<char,std::char_traits<char>,std::allocator<char> > modopt(IsConst)* modopt(IsConst) modopt(IsConst)) localPtr));
        }
        fault
        {
            ___CxxCallUnwindDtor(std.basic_string<char,std::char_traits<char>,std::allocator<char> >.{dtor}, (void*) &local);
        }
        std.basic_string<char,std::char_traits<char>,std::allocator<char> >.{dtor}(&local);
    }
    fault
    {
        ___CxxCallUnwindDtor(std.basic_string<char,std::char_traits<char>,std::allocator<char> >.{dtor}, (void*) &local2);
    }
    std.basic_string<char,std::char_traits<char>,std::allocator<char> >.{dtor}(&local2);
    return str;
}

I notice it also references this, but I don't understand what it does at all...

[PreserveSig, MethodImpl(MethodImplOptions.Unmanaged, MethodCodeType=MethodCodeType.Native), SuppressUnmanagedCodeSecurity]
internal static unsafe basic_string<char,std::char_traits<char>,std::allocator<char> >* modreq(IsUdtReturn) modopt(CallConvCdecl) _Encrypt(basic_string<char,std::char_traits<char>,std::allocator<char> >*, basic_string<char,std::char_traits<char>,std::allocator<char> > modopt(IsConst)* modopt(IsImplicitlyDereferenced));

I'm still new to desktop programming, my only programming experience is mostly with web-based technologies and the absolute basics of C++. I've never gotten into reverse engineering before either. So please be gentle and thorough in your response as I'd really like to learn.

-------------- EDIT

I've finally been able to find the same function call in OllyDBG. However, setting a breakpoint and making it run the code does not cause it to break.

What might I be doing wrong? :/

CPU Disasm
Address   Hex dump          Command                            Comments
00A088D8  /$  02            ldarg.0
00A088D9  |.  7B CC000004   ldfld openFileDialog3
00A088DE  |.  02            ldarg.0
00A088DF  |.  6F D101000A   callvirt ShowDialog
00A088E4  |.  17            ldc.i4.1
00A088E5  |.- 2E 01         beq.s 0A088E8
00A088E7  |.  2A            ret
00A088E8  |>  02            ldarg.0
00A088E9  |.  7B CC000004   ldfld openFileDialog3
00A088EE  |.  6F 8501000A   callvirt get_FileName
00A088F3  |.  28 D201000A   call ReadAllText
00A088F8  |.  0A            stloc.0
00A088F9  |.  06            ldloc.0
00A088FA  |.  6F 9000000A   callvirt Trim
00A088FF  |.  6F 3900000A   callvirt get_Length
00A08904  |.- 2D 01         brtrue.s 0A08907
00A08906  |.  2A            ret
00A08907  |>  06            ldloc.0
00A08908  |.  6F 9000000A   callvirt Trim
00A0890D  |.  72 99340070   ldstr "ENCRYPTED"                  ; UNICODE "ENCRYPTED"
00A08912  |.  6F D301000A   callvirt StartsWith
00A08917  |.- 2C 1F         brfalse.s 0A08938
00A08919  |.  02            ldarg.0
00A0891A  |.  72 AD340070   ldstr "File is already encrypted"  ; UNICODE "File is already encrypted"
00A0891F  |.  28 14000006   call Translate
00A08924  |.  72 E1340070   ldstr "Error"                      ; UNICODE "Error"
00A08929  |.  28 14000006   call Translate
00A0892E  |.  16            ldc.i4.0
00A0892F  |.  1F 10         ldc.i4.s 10
00A08931  |.  28 D401000A   call Show
00A08936  |.  26            pop
00A08937  |.  2A            ret
00A08938  |>  06            ldloc.0
00A08939  |.  28 D501000A   call Encrypt
00A0893E  |.  0A            stloc.0
00A0893F  |.  06            ldloc.0
00A08940  |.  6F 9000000A   callvirt Trim
00A08945  |.  6F 3900000A   callvirt get_Length
00A0894A  |.- 2D 01         brtrue.s 0A0894D
00A0894C  |.  2A            ret
00A0894D  |>  02            ldarg.0
00A0894E  |.  7B CC000004   ldfld openFileDialog3
00A08953  |.  6F 8501000A   callvirt get_FileName
00A08958  |.  02            ldarg.0
00A08959  |.  7B CC000004   ldfld openFileDialog3
00A0895E  |.  6F 8501000A   callvirt get_FileName
00A08963  |.  72 ED340070   ldstr ".bak"                       ; UNICODE ".bak"
00A08968  |.  28 3A00000A   call Concat
00A0896D  |.  17            ldc.i4.1
00A0896E  |.  28 B700000A   call Copy
00A08973  |.  02            ldarg.0
00A08974  |.  7B CC000004   ldfld openFileDialog3
00A08979  |.  6F 8501000A   callvirt get_FileName
00A0897E  |.  06            ldloc.0
00A0897F  |.  28 A200000A   call WriteAllText
00A08984  |.  02            ldarg.0
00A08985  |.  72 F7340070   ldstr "File encrypted"             ; UNICODE "File encrypted"
00A0898A  |.  28 14000006   call Translate
00A0898F  |.  72 15350070   ldstr "Info"                       ; UNICODE "Info"
00A08994  |.  28 14000006   call Translate
00A08999  |.  16            ldc.i4.0
00A0899A  |.  1F 40         ldc.i4.s 40
00A0899C  |.  28 D401000A   call Show
00A089A1  |.  26            pop
00A089A2  \.  2A            ret

Answer 1

[PreserveSig, MethodImpl(MethodImplOptions.Unmanaged, MethodCodeType=MethodCodeType.Native), SuppressUnmanagedCodeSecurity]

You see the first part saying unmanaged? .net is a 'managed' environment. As this is referred to the memory, I can tell you C and C++ are not (beware of C++.net as that is managed C++.

Anyway, by the looks of this I'd say it calls the function from an external source (I'd suspect a DLL).

internal static unsafe basic_string<char,std::char_traits<char>,std::allocator<char> >* modreq(IsUdtReturn) modopt(CallConvCdecl) _Encrypt(basic_string<char,std::char_traits<char>,std::allocator<char> >*, basic_string<char,std::char_traits<char>,std::allocator<char> > modopt(IsConst)* modopt(IsImplicitlyDereferenced));

This says things about the method it is called Cdecl the function is called _Encrypt and has a few arguments.

Hope this helped, now you should first learn the basics, especially when you are going to dive into disassembling this native code.