0

I have a data set where events have a unique ID. Some examples:

CkUKGkNNZmY1TkhObi1zQ0ZWT0h3Z0VkaExJQ0xnEidDSlB0bG9UTG4tc0NGWmhpaXdvZDFxOE5fdzE1OTc1NzY4NzU4Mzg%3D
CjoKGkNOWHBsX1hJbi1zQ0ZTWks3UW9kRDRNSVdREhxDSXI4emVuSW4tc0NGZm5WVEFJZEVRZ0JrUS0x
CjkKGkNMTzJ6TkRWbi1zQ0ZUNkh3Z0VkdGVVQnd3EhtDT3ZOajZYVm4tc0NGWUdGY0FvZGJIc09iZzE%3D
CjkKGkNMcnN4OHZKbi1zQ0ZRMXpnd29kUW9VQll3EhtDSTdfZzVmSm4tc0NGVWFBY0FvZGRsQUhFQTQ%3D
CjsKGkNMNjlrdFRObi1zQ0ZZMkN3Z0VkVWxnSGxREh1DSktpanFISm4tc0NGVHZCVEFJZEM3a01Udy00OA%3D%3D
CjkKGkNNQ2ItbzNWbi1zQ0ZkU013Z0VkX2RRRHNBEhtDSlAzcm8zUm4tc0NGV252T0FZZENkUUJyUTU%3D
CjoKGkNOTzhoZVRRbi1zQ0ZVYTBnZ29kN0VrTnlnEhxDSS1qbmRmUW4tc0NGY1pLaXdvZEgxRUFvUS0w
CkUKGkNQU0VocjNMbi1zQ0ZaaXNnZ29kQlNjTTZBEidDTVNmbDRiTG4tc0NGWDRBdHdBZF9lQUtPQTE1OTc1NzYyOTkyNDg%3D
CjkKGkNJLVh3NEhLbi1zQ0ZaV3JEUW9kUzNBSC1BEhtDTmFWeVpuSm4tc0NGZEtSY0FvZFdWNEVvQTE%3D
CjkKGkNKYnZpb3pSbi1zQ0ZVVkE3UW9kV0xzTnVBEhtDSXlKcktQUW4tc0NGZkhEVEFJZEJtOENGdzQ%3D
CjsKGkNJRGZsZnpVbi1zQ0ZjM3hnZ29kd1hZRnVREh1DS0t1bE96VW4tc0NGUlQ1V0FvZGE2OElSQS0yNg%3D%3D
CkUKGkNJVGhuZHpQbi1zQ0ZZMkJ3Z0VkUkk0Q1N3EidDUC1qbDVITm4tc0NGVnBsaXdvZGJ2b0hLUTE1OTc1Nzc0MzU0Mjg%3D
CkUKGkNPRzc4SkRPbi1zQ0ZkbWhnZ29kckprTWJREidDT2FCdmRUTW4tc0NGYnhDOVFVZExfc0lIQTE1OTc1NzcwMDk0NTQ%3D
CjoKGkNJN05fZURWbi1zQ0ZZRzJEUW9kamEwTXJ3EhxDTHJ5eVpMVm4tc0NGVWZDVEFJZEkyNEdIQS00
CkUKGkNNN1F6TERKbi1zQ0ZWbUd3Z0Vkb0RJSzlREidDT3ZBeUszSm4tc0NGU1FfWUFvZEl1UUZ4dzE1OTc1NzU3MzIyNTM%3D
CjkKGkNJeXlwOV9Kbi1zQ0ZkQzZnZ29kaGdBSWVnEhtDSVdRMzd6Qm4tc0NGWmlDckFJZEFNb05BZzU%3D
CjoKGkNKQ1YzWnpUbi1zQ0ZjWW5nd29kaUVJRkdREhxDSXVieHBqU24tc0NGVUZrS2dvZElRNElndy0x
CjkKGkNMVHJrNWpMbi1zQ0ZVa29nd29kSG1FRmtnEhtDS3VXdGRMS24tc0NGUVdIY0FvZFFuWUVfdzA%3D
CjkKGkNQVDhqOV9Tbi1zQ0ZaUlU3UW9kSGZZQkl3EhtDTV9rak9iUG4tc0NGUkVsandvZFdIa1A5QTg%3D
CkUKGkNKdmM4S0xRbi1zQ0ZSaktnZ29kdWdRTm5REidDS096NDRQTm4tc0NGUXQ3aXdvZDI1Z0c4dzE1OTc1Nzc1ODMwMDM%3D
CjoKGkNOM3ZnTXJQbi1zQ0ZaSk03UW9kWmJvQmx3EhxDSUdVcHBMSm4tc0NGWVptTUFvZFg3SVBZUTMy
CjkKGkNKYjZ6YmZMbi1zQ0ZRNkV3Z0VkaW9FRDZREhtDTV9UcG96TG4tc0NGWWpGY3dFZExiVUp0ZzA%3D
CjkKGkNPRHZ1Yl9Wbi1zQ0ZZeGU3UW9kMzZ3UG5nEhtDTmYzcFp2R24tc0NGUlpUandvZEt3SUswUTk%3D
CjkKGkNKcjV3NkRKbi1zQ0ZUNkh3Z0VkT1RZSGtBEhtDUFRJeXB6Sm4tc0NGZmp5T0FZZENqZ1BfZzA%3D
CkUKGkNOZlM4NlhKbi1zQ0ZRcWhnZ29ka3kwUEt3EidDSVBwcDVUSm4tc0NGZl9GVEFJZGhIa0ZiZzE1OTc1NzU3MDc1Njk%3D
CjkKGkNONmNsZmJKbi1zQ0ZkYTlnZ29kU3cwRnFBEhtDUEMyOEtMSG4tc0NGZkdDWXdZZDgtTUprUTk%3D
CkUKGkNLZTJ1TmpSbi1zQ0ZTTGdnZ29kcDJVSkdBEidDTW1YcHF2Um4tc0NGUmhQS2dvZGRPQURUdzE1OTc1Nzc4ODk4NjM%3D
CjoKGkNOang2NUxUbi1zQ0ZRSlE3UW9kN1JNRUlREhxDSlNRaDZqRm4tc0NGWFJSaFFvZFJmUUlZZzgx
CkUKGkNLT2dpYXZVbi1zQ0ZRV21EUW9kTngwRy1BEidDSm5lZ3FEUm4tc0NGUzVDaFFvZHlxb0V4dzE1OTc1Nzg2NzI5NjI%3D
CkUKGkNLaU8tZURKbi1zQ0ZlcUZ3Z0VkRGhrT1V3EidDTFdEcUtISm4tc0NGYXhDaFFvZFlSRUNmUTE1OTc1NzU4MzM3NDM%3D
CkUKGkNPMkJ1YlBLbi1zQ0ZlUGxnZ29kN3JvUFZBEidDTWlfdlozSm4tc0NGWVBKV0FvZDZwb05hQTE1OTc1NzYwMDY3ODk%3D
CkUKGkNKR09xS1hTbi1zQ0ZZUkM3UW9kV1FvRzh3EidDUFc2dEkzU24tc0NGUlI0WUFvZGdHb0ZKUTE1OTc1NzgxMjQ0MjY%3D
CjoKGkNNekd1N2JObi1zQ0ZjeXlnZ29kdmJZTXZBEhxDTnlnN003S24tc0NGVVZDOVFVZFktZ0Zwdy02
CjoKGkNPemd2cmJPbi1zQ0ZZTnRnd29kbzFvSy1nEhxDTEtMMnFfSm4tc0NGWmRBOVFVZGR5WUVrUTEz
CjoKGkNLbUxxZXpRbi1zQ0ZaUE1nZ29kbnAwSDRBEhxDTmVqbE96SW4tc0NGUTlIV0FvZDV5QUdaQTEy
CjsKGkNKcjMzSkhLbi1zQ0ZSaktnZ29kZ2RnS3N3Eh1DT1N3eXBESm4tc0NGYXhTaFFvZExJSUdlUS0xMw%3D%3D
CkUKGkNKT3luXzdSbi1zQ0ZhSkQ3UW9kLWlRTHNnEidDSUhnaVlqUm4tc0NGWVZCOVFVZFdqc05JQTE1OTc1NzgwNDM3MzE%3D
CjoKGkNQbWM5UHpQbi1zQ0ZSblBnZ29kTlU0Q3d3EhxDTXVEa3RqS24tc0NGZDVBOVFVZE43MEN1dzIz
CkUKGkNMX3k5UDdUbi1zQ0ZRU0d3Z0VkQ2tvR2t3EidDSXVqb19IVG4tc0NGU0pEOVFVZGRZMERyZzE1OTc1Nzg1ODAyODI%3D
CjkKGkNJSGJ0b2pMbi1zQ0ZXSDNnZ29kLW5vTlpREhtDSjNrMV8zS24tc0NGUWJ3T0FZZHlJa01rZzA%3D
CjoKGkNKT2I4ZHpObi1zQ0ZjcU13Z0VkWjkwRkpBEhxDTUtsc01uTW4tc0NGVnpPVEFJZERmWU9idy0z
CkUKGkNNSFVtS19Tbi1zQ0ZjT0J3Z0VkcHRVSXBnEidDTnVLbk9mUm4tc0NGYzVQS2dvZG53WURmUTE1OTc1NzgxNDQ2ODg%3D
CjkKGkNNYXJnS1BKbi1zQ0ZRcVJ3Z0VkemdzSGlREhtDT1RCcTdQQm4tc0NGVWIwT0FZZFFwSUpFUTM%3D
CkUKGkNJcnFqcGZLbi1zQ0ZVMkh3Z0VkOVpvSHB3EidDS2UyaGZ6Sm4tc0NGWnJIVEFJZDM4a1BjUTE1OTc1NzU5NDY4OTI%3D
CjkKGkNJdkM4bzdTbi1zQ0ZaV1BnZ29kWWN3SXd3EhtDTGlfby1MSm4tc0NGUXBCaFFvZDMzQU9wZzA%3D
CkUKGkNKYXRqc1RSbi1zQ0ZaQ2xnZ29kTTAwRGpBEidDSW5mejk3S24tc0NGWVJEaFFvZElNd043dzE1OTc1Nzc5MjE1NTQ%3D
CkUKGkNMSEgyZGpUbi1zQ0ZZVDBnZ29kOGRzR0tREidDT2lqdTRyUm4tc0NGWWRKWUFvZFg4TU8xdzE1OTc1Nzg1MDA5NDY%3D
CjoKGkNLQzN3cjNWbi1zQ0ZRRy1nZ29kQXMwSW5REhxDSXV5NXBuR24tc0NGUXotT0FZZHB6a0h5UTE1
CjkKGkNNXzZpWkRObi1zQ0ZZTHdnZ29kZ0JFRGlREhtDTlRWNElIS24tc0NGVXB3andvZHRoMENfZzQ%3D
CjoKGkNMNk56dXZQbi1zQ0ZSU3NEUW9kUnRjSGtREhxDTGF0c05yTW4tc0NGUVBoV0FvZHY5Z0xNUS05
CjkKGkNQVzJsTFRNbi1zQ0ZYNkh3Z0VkY25FTG53EhtDT21CajZmTW4tc0NGZE9CY0FvZHF0b0tsZzA%3D

Starting from the surface, they seem to

  1. be URL encoded
  2. be base64 encoded (although there are only 62 unique characters and = padding)
  3. have some struct that contains two URL safe base64 strings https://tools.ietf.org/html/rfc4648#section-5
    • the format is \n<length byte>\n<length byte><first string>\x12<length byte><second string>

I know what the second string points to, but I'm interested in the first one.

Everything that I've seen so far matches this pattern:

grp1, grp2, grp3 = re.match(b'^\x08(.{7})\x02\x15(.{4})\x1d(.{4})$', part1_decoded, re.DOTALL).groups()

grp1 can have 128 unique characters, but grp2 and grp3 can have all 256. However, some characters are more common than others.

  • In grp1 the most common ones (by a large margin) are 9f and eb. There's also a spike between c9 and d5. 9f and eb are usually on the right side, so it might be a sequential little-endian integer such as a database index or a timestamp.
  • In grp2, characters like 01 \n \r 82 c2 are common. I can't make anything out of it except that they could be separators.
  • grp3 looks otherwise random except characters from 00 to 0f are ~8 times more common than others.

I'm looking for any clues or analysis methods to find out what groups 1, 2 and 3 are used for.

2
  • do you have the code which generates or parses them? – Igor Skochinsky Oct 12 '20 at 15:18
  • @IgorSkochinsky This script does something with them, but I believe it doesn't parse them and only uses them to identify the events youtube.com/yts/jsbin/live_chat_polymer-vflyLUH5M/… . At least this.data.id is probably the ID here. The only code I know to parse the ID is my own. – siikamiika Oct 12 '20 at 16:34
0

The entire thing seems to be protobuf after doing the initial unwrapping with urldecode and base64 decode. grp1 was a timestamp, but I'm not yet sure what the 32-bit floats are used for. Should be easier to go forward now that the format is known, though.

I found this tool helpful decoding the data: https://github.com/omarroth/protodec

> echo 'CkUKGkNNZmY1TkhObi1zQ0ZWT0h3Z0VkaExJQ0xnEidDSlB0bG9UTG4tc0NGWmhpaXdvZDFxOE5fdzE1OTc1NzY4NzU4Mzg%3D' | php -r 'echo urldecode(fgets(STDIN));' | ./protodec -bp | jq
{
  "1:0:embedded": {
    "1:0:base64": {
      "1:0:varint": 1597576876470215,
      "2:1:float32": 7.14585257493996e-38,
      "3:2:float32": 2.971713153332445e-11
    },
    "2:1:string": "CJPtloTLn-sCFZhiiwod1q8N_w1597576875838"
  }
}

# the left part of the above string
> echo 'CJPtloTLn-sCFZhiiwod1q8N_w' | ./protodec -bp | jq
{
  "1:0:varint": 1597576176842387,
  "2:1:float32": 1.3422299960259732e-32,
  "3:2:float32": -1.883341397915719e+38
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.