Figuring out encryption technique of network connection

Question

I am trying to reverse a virtualized program (using VMProtect 3.4.x). I decided to hook certain functions of the program such as the winsocks library send() and recv().

This is how it looks like:

[monitor] send() buf: ╣▒√≈o└┼└1£ len: 12 flags: 0
[monitor] recv() buf: ╣▒√≈o└═└1£ len: 12 flags: 0
[monitor] recv() buf: ï|W,7¼≤╛ len: 8 flags: 0
[monitor] send() buf: √X╥(£!◄▀√εPî'Zë¬08ꣽ╘╤µ0ߺÉ╥╬=S3τU_╜µⁿºi┼²f░uw╦╖TX▓°┼ùΘ¿╡ len: 58 flags: 0
[monitor] recv() buf: √X╥(₧!◄▀╘εPî len: 12 flags: 0
[monitor] recv() buf: % len: 1 flags: 0
[monitor] send() buf: √X╥(Æ!◄▀òεPîT♂╥πc_╨ÿ╒╜ùΓzï╟ÿ▐Ω§Sc┘9~ïσµ┤P╟σRà⌂X╚éT2╪²µ╛Γ╓π≡3r~╗l╘O╓▀]PÖån╟ǽ len: 76 flags: 0
[monitor] recv() buf: √X╥(₧!◄▀╘εPî len: 12 flags: 0
[monitor] recv() buf: % len: 1 flags: 0

Another run:

[monitor] send() buf: ╣▒√≈o└┼└1£ len: 12 flags: 0
[monitor] recv() buf: ╣▒√≈o└═└1£ len: 12 flags: 0
[monitor] recv() buf: ïxS*4¬≤╗ len: 8 flags: 0
[monitor] send() buf: F►²cgTα2╬|4÷«ÄY░Θs*Bít╡Yl⌠rp↑↑─╨¶WD┬☼«U;÷ƒçz▼╣┐┌n╘Γ│óHUU len: 58 flags: 0
[monitor] recv() buf: F►²ceTα2ß|4÷ len: 12 flags: 0
[monitor] recv() buf: ¼ len: 1 flags: 0
[monitor] send() buf: F►²ciTα2á|4÷▌▀☻∙║¶rF▀↔≤]&₧↕x¶<∞╨Di(π9¡O(╧¥ƒN2§û╝∩n╛ê╢$ïC+♥σ╠n▄E÷sÿ↑¬╗z°☻═a] len: 76 flags: 0
[monitor] recv() buf: F►²ceTα2ß|4÷ len: 12 flags: 0
[monitor] recv() buf: ¼ len: 1 flags: 0

I would like to know what is being send and received. I don't know how the final buffer will look like. Given the fact that there are some repeating symbols I don't think any complex algorithm is used.

Is there any (somewhat) easy way to figure it out or do I have to go though the virtualized code?

Answer 1

For static analysis, you can use VMAttack plugin for IDA pro. Or if you wanna find how the buffer is built before it's sent you can try ScyllaHide plugin for x64dbg to debug the VmProtected executable. This plugin will help you overcome VMprotect's anti-debug and anti-vm protections. After that simply put a breakpoint on those calls that you hooked and follow the execution flow to figure out how the encryption works. In either case you'll need to deal with the virtualized instructions

Answer 2

Post the data as hex. 100 messages would be great.

Are you using ANSI encoding to make patterns in the data easier to see?

My guess is there's something like ROT or an xor going on here with messages [monitor] recv() buf: ï|W,7¼≤╛ and [monitor] recv() buf: ïxS*4¬≤╗ being the key the server wants the client to use as that's the first place they change between runs. I've broken apart the messages based on where values change between messages. Those would be places where either the length of the message is indicated or the function of the message is indicated. For example (£,₧) and (√,╘).

You can take the message length for the variable length parts and that will get you the values for £ or √ and Æ or ò subject to some common transform. That should be the path to figuring out how it's encoded.

[monitor] send() buf: ╣▒√≈o└    ┼   └1£ 
[monitor] recv() buf: ╣▒√≈o└    ═   └1£ 

[monitor] recv() buf: ï |W,7¼≤╛ 

[monitor] send() buf: √X╥(  £   !◄▀ √   εPî     'Zë¬08ꣽ╘╤µ0ߺÉ╥╬=S3τU_╜µⁿºi┼²f░uw╦╖TX▓°┼ùΘ¿╡                  
[monitor] recv() buf: √X╥(  ₧   !◄▀ ╘   εPî                                                                     

[monitor] recv() buf: %                                                                                 

[monitor] send() buf: √X╥(  Æ   !◄▀ ò   εPî     T♂╥πc_╨ÿ╒╜ùΓzï╟ÿ▐Ω§Sc┘9~ïσµ┤P╟σRà⌂X╚éT2╪²µ╛Γ╓π≡3r~╗l╘O╓▀]PÖån╟ǽ 
[monitor] recv() buf: √X╥(  ₧   !◄▀ ╘   εPî                                                                     

[monitor] recv() buf: %                                                                                 



[monitor] send() buf: ╣▒√≈o└    ┼   └1£                                                                     
[monitor] recv() buf: ╣▒√≈o└    ═   └1£                                                                         

[monitor] recv() buf: ï xS*4¬≤╗ 

[monitor] send() buf: F►²c  g   Tα2 ╬   |4÷     «ÄY░Θs*Bít╡Yl⌠rp↑↑─╨¶WD┬☼«U;÷ƒçz▼╣┐┌n╘Γ│óHUU 
[monitor] recv() buf: F►²c  e   Tα2 ß   |4÷ 

[monitor] recv() buf: ¼ 

[monitor] send() buf: F►²c  i   Tα2 á   |4÷     ▌▀☻∙║¶rF▀↔≤]&₧↕x¶<∞╨Di(π9¡O(╧¥ƒN2§û╝∩n╛ê╢$ïC+♥σ╠n▄E÷sÿ↑¬╗z°☻═a] 
[monitor] recv() buf: F►²c  e   Tα2 ß   |4÷ 

[monitor] recv() buf: ¼