Goal: Trying to reverse encrypter to decrypter that successfully decrypts.

This is the pseudo-code of the disassembled encrypting program. There are no function calls, and I've been able to somewhat replicate this code in C.

undefined8 main(void)

  int random;
  time_t current_time;
  long in_FS_OFFSET;
  uint time_number;
  uint rand_seven;
  long counter;
  FILE *open_flag;
  size_t size_of_flag_file;
  void *pointer_allocated_memory_block;
  FILE *open_flag_enc;
  long off_40;
  off_40 = *(long *)(in_FS_OFFSET + 40);
  open_flag = fopen("flag","rb");
  size_of_flag_file = ftell(open_flag);
  pointer_allocated_memory_block = malloc(size_of_flag_file);
  current_time = time(NULL);
  time_number = (uint)current_time;
  for (counter = 0; counter < (long)size_of_flag_file; counter = counter + 1) {
    random = rand();
    *(byte *)((long)pointer_allocated_memory_block + counter) = *(byte *)((long)pointer_allocated_memory_block + counter) ^ (byte)random;
    rand_seven = rand();
    rand_seven = rand_seven & 7;
    *(byte *)((long)pointer_allocated_memory_block + counter) = *(byte *)((long)pointer_allocated_memory_block + counter) << (sbyte)rand_seven | *(byte *)((long)pointer_allocated_memory_block + counter) >> 8 - (sbyte)rand_seven;
  open_flag_enc = fopen("flag.enc","wb");
  if (off_40 != *(long *)(in_FS_OFFSET + 40)) {
                    /* WARNING: Subroutine does not return */
  return 0;

This is the code that needs work: I have tried multiple iterations. If I could get some help or an overview of what I could be/am missing?

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main() {
    unsigned int time_number;
    size_t counter;
    FILE *open_flag;
    size_t size_of_flag_file;
    void *pointer_allocated_memory_block;
    FILE *open_flag_enc;

    printf("[DEBUG] Starting program...\n");

    // Open the encrypted file
    printf("[DEBUG] Attempting to open encrypted file...\n");
    open_flag = fopen("flag.enc", "rb");
    if (open_flag == NULL) {
        perror("[ERROR] Could not open file");
        return 1;
    printf("[LOG] Successfully opened encrypted file.\n");

    // Read the RNG seed from the first 4 bytes of the file
    fread(&time_number, 1, 4, open_flag);
    printf("[LOG] RNG Seed read: %u\n", time_number);

    // Determine the size of the remaining file
    fseek(open_flag, 0, SEEK_END);
    size_of_flag_file = ftell(open_flag) - 4;
    fseek(open_flag, 4, SEEK_SET);
    printf("[LOG] Size of remaining file: %zu bytes\n", size_of_flag_file);

    // Read the encrypted flag into memory
    pointer_allocated_memory_block = malloc(size_of_flag_file);
    fread(pointer_allocated_memory_block, 1, size_of_flag_file, open_flag);
    printf("[LOG] Successfully read encrypted data into memory.\n");

    // Seed RNG with the same seed
    printf("[DEBUG] Seeding RNG...\n");

    // Decrypt the flag
    printf("[DEBUG] Starting decryption...\n");
    unsigned char *buffer = (unsigned char *)pointer_allocated_memory_block;
    for (counter = 0; counter < size_of_flag_file; counter++) {
        int random = rand();
        int rand_seven = rand() & 7;
        buffer[counter] = (buffer[counter] << (8 - rand_seven)) | (buffer[counter] >> rand_seven);
        buffer[counter] ^= (unsigned char)random;
    printf("[LOG] Decryption completed.\n");

    // Write the decrypted flag to "flag.txt"
    printf("[DEBUG] Writing decrypted flag to file...\n");
    open_flag_enc = fopen("flag.txt", "wb");
    fwrite(pointer_allocated_memory_block, 1, size_of_flag_file, open_flag_enc);
    printf("[LOG] Successfully wrote decrypted flag to 'flag.txt'.\n");

    // Free allocated memory
    printf("[DEBUG] Freeing allocated memory...\n");

    printf("[LOG] Program completed successfully.\n");
    return 0;

Thank you for your time.

  • 1
    It looks like you're initializing srand with a constant in your decriptor, while in encryptor it is initialized with something else. Are you sure that it should be so?
    – w s
    Commented Aug 31, 2023 at 15:30
  • Previous comment is correct. The decompilation shows that the seed for the random number generator is the first 4 bytes in the .enc file. You're using a hard-coded seed, though. You need to read the first 4 bytes and use that as your seed instead of using a fixed value. You appear to have realized that, and then commented the code out and used a fixed seed anyway. Commented Aug 31, 2023 at 16:20
  • Yes, I was initially trying something with the original program and comparing it. I have it fixed to the first 4 bytes of the file now. I'll edit the post to reflect that. Commented Aug 31, 2023 at 17:25
  • So the problem is you don't get what you expect when your decryptor runs? I'd recommend putting debug prints at each iteration in your decryption loop, and then stepping through the original program in a debugger and comparing the values along the way. The problem might be that the C standard library you're using implements rand (and srand) differently from the binary's version. The validation process I outlined will let you hone in on the problem very quickly. Commented Aug 31, 2023 at 22:03
  • Thank you, I didn't think to compare the standards and libraries. That could be the problem. I'll try out that validation process and compare and fix any libraries. I'll let you know how it goes. Commented Sep 1, 2023 at 1:08

1 Answer 1


I'd recommend debugging the decryption loop in your code, and the encryption loop in the original binary, side-by-side and comparing the values at each step. It's possible that your binary's rand() might be producing different values from the one in the original binary -- the behavior of rand() may vary by version of the C standard library, or across implementations of the C standard library. If that's the case, you'd want to replicate their implementation, or just try to use the same standard library that they did.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.