I have a function that takes some arguments and returns a char array which is being used by send() function later on (basically a packet encoder). My initial approach was to try and decompile this big encode function but there're at least 20-40 sub functions also data types are not really easy to figure out so i gave up on that. My second approach was to try to use hex-rays decompiler plugin but that also gives some uncompilable c code. As last resort i tried to analyze input data and encoded data but since the input changes all the time during runtime it's really hard to compare results and the only conclusion i could reach was first byte is a header and next two bytes represent packet length. So my question is, is there any way to feed data into a function(subroutine) with IDA Pro or any other disassembly tool ?
Yes, you can use Appcall: https://hex-rays.com/products/ida/support/tutorials/debugging_appcall.pdf
Appcall is a mechanism to call functions inside the debugged program from the debugger or your script as if it were a built-in function.