I have a customized DLL in a large software. The DLL compress/decompress image files. I need to use the DLL separately to get my important images back to decompressed version. I can see its function in IDA pro, but I am not sure whether my finding is correct. And I don't know how to call the function properly in C#. The attached pictures is my IDA pro result and my test C# program(of course the program doesn't work). Please help me.

DLL function in IDA Pro

Test C# program

  • 1
    Why are you trying to return byte[]? The disassembled code seems to return FILE*. – svick Apr 23 '16 at 13:41
  • Did you try debugging the original DLL to see what are the real parameters? Also, I'd recommend first interfacing using C, to be sure the interfaces are ok before dealing with the possible marshaling issues. – tmr232 Apr 23 '16 at 19:16

First of all, follow the advice of the two commentors and investigate on the semantics of the ArcCompress method. In case it is really returning a C file pointer, what would you do with it in C# ?? Not much, I assume. You wouldn't probably be able to close the file there.

But anyway, here are two possibilities to call your ArcCompress method from C#. As a small test, first a fictive simple ArcCompress C dll function, as a test facility to see whether the pointers are marshalled correctly. The function must be declared as extern "C" to avoid name mangling.

//Next line is from the header file
//#define PINVOKE_DLL_API __declspec(dllexport)
PINVOKE_DLL_API FILE* __cdecl ArcCompress(char* a1, char* a2, int a3)
    a1[2] = 0xfe;
    FILE* f = fopen("D:\\testfile.bin", "wb+");
    fwrite(a1, a3, 1, f);
    return f;

This simple function accepts your byte arrays, modifies one byte and writes it into a file, returning the file pointer of the open file. And here come two possibilities to call this function from C# (In my example, the dll is named "pinvoke_dll.dll" instead of "Arc.dll"):

  1. The buffer parameters are ordinary references, not much special here. It should be checked however if there may arise problems with the .NET garbage collector. The arrays are assumed as having been initialized.

    [DllImport("pinvoke_dll.dll", CallingConvention = CallingConvention.Cdecl)]
    public static extern IntPtr ArcCompress(ref byte a,  ref byte b, Int32 c);
    private void button1_Click(object sender, EventArgs e)
        IntPtr f = ArcCompress(ref a[0], ref b[0], a.Length );
  2. The buffer parameters are C# pointers being treated just like ordinary C-pointers. In this case you must tag your method as "unsafe" (and check the corresponding checkbox in the project properties), and mark the pointers as fixed, thus informing the garbage collector not to touch them.

public unsafe partial class Form1 : Form { public Form1() { InitializeComponent(); } [DllImport("pinvoke_dll.dll", CallingConvention = CallingConvention.Cdecl)] public static extern IntPtr ArcCompress(byte* p1, byte* p2, Int32 c); byte[] a = new byte[10]{1,2,3,4,5,6,7,8,9,0}; byte[] b = new byte[10]{0,9,8,7,6,5,4,3,2,1}; private void button1_Click(object sender, EventArgs e) { fixed (byte* p1 = &a[0], p2 = &b[0]) { IntPtr f = ArcCompress(p1, p2, a.Length); } } } BTW, in this way you may apply pointer arithmetics in C# just like in C. You must have in mind that you are in an unsafe context, just like in C or C++.

Have fun!

| improve this answer | |
  • Thank you for your answer. My problem is solved. The function actually takes two file name strings and returns an int The correct function call is in c# is [DllImport("Arc.dll", CallingConvention = CallingConvention.Cdecl, EntryPoint = "ArcCompress")] public static extern int ArcCompress(byte[] infile, byte[] outfile, ref int method); – xken629 Apr 28 '16 at 2:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.