Identifying the source of encryption used by UDP packets in a PCAP file

Question

• I have a PCAP file (mediafire link to the file) which basically represents packet captures between 2 machines running the same game connected to each other via LAN inside RPCS3 using RPCN.

• One of them has the inet address 192.168.0.104 and the other one is 192.168.0.100.

• Many UDP packets are being transmitted between both which I believe is the game data.

• If you filter the packets by "tls" there are 2 TLS handshakes of which I have been able to decrypt the first one using session keys.

• It has data related to RPCS3 connection.

• My question is that are the UDP packets encrypted with the keys generated in the 2nd handshake or are they using some custom encryption that the game itself manages in its code?

• The second SSL handshake mostly(90% sure) represents the act of logging into the RPCN network inside RPCS3.

• How come there are no SSL handshakes between the 2 LAN machines?

Answer 1

To your first question, you'd have to look at the code. There may be encryption, but some of the UDP packets have very small payloads, and using encryption would slow down packet processing which is the opposite of the key purpose for using UDP.. which is speed.

For your second question, the LAN game is using UDP, and TLS needs TCP. If you think about the use case.. TLS/SSL is to verify that at least one end of the connection is who it says it is. If you're on the LAN, you don't know who is who (no DNS) and you don't care.. TLS makes no sense in this case.

I think you're correct that the TLS connections are being used during user authN to the game servers. However, given that the session keys negotiated between 192.168.0.104 <--> game server are used to secure only communication between those 2 hosts, they would not be used between 192.168.0.104 <--> 192.168.0.100 on the LAN as they provide no value. They do not identify either host, and they slow down game performance.

This is a little wordy but I hope it helps.