I'm working my way through reversing a toy challenge, and I find myself stuck. The app is pretty simple, it spits out a blob of text (e.g. "3b880a90e476d66569d9d5dfb5cd755af3f..."). Dumping the code, I can see that it builds an RSA public key by directly specifying n and e:


Then it encrypts it's payload:

encodedLength = RSA_public_encrypt(flen, from, to, myRsa, 1);

My goal: steal the payload. Debugging tells me flen is 240, encodedLength is 100. I dumped the e for the public key as bytes, and wrote some code to generate my own RSA public/private key, patching e to be mine (n is the same for both, so left unpatched).

unsigned long bytes_read = fread(in, sizeof(unsigned char), size, file);

    for (size_t i = 0; i < bytes_read; i++) {
        if (memcmp(nCharOrig, in + i, 258) == 0) {
            memcpy(in + i, nCharNew, 258);
            printf("Found and patched at %d\n", i);

Patch works, I get a different blob of text, which in theory is the same payload, encrypted with my public key. So I try to decrypt it:

unsigned char output[8000];
    RSA *rsa = RSA_new();
    EVP_PKEY *privkey;
    FILE *fp;

    privkey = EVP_PKEY_new();
    fp = fopen ("private.pem", "r");
    PEM_read_PrivateKey( fp, &privkey, NULL, NULL);

    rsa = EVP_PKEY_get1_RSA(privkey);
    int decryptLength = RSA_private_decrypt(256, input, output, rsa, 1);

Weirdly I get back decryptLength = -1, and an error: error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error

I'm at a loss.. what am I missing here?

  • Have you confirmed that the keys you generated are valid? (openssl verify) Jan 12, 2023 at 23:52
  • Can you share your toy challenge? Also whats the value of e?
    – sudhackar
    Jan 13, 2023 at 5:51
  • If you're able to patch the app, is there a reason you can't patch it to dump the unencrypted payload instead? Either that or get the unencrypted payload via the debugger? Jan 13, 2023 at 16:07
  • 1
    @hairlessbear I actually did steal it via debugger, that was easy :) Doing it this way as a challenge to myself to finally understand openssl/RSA code reversing correctly. Jan 13, 2023 at 17:52
  • @MegaTonnage yep, wrote them out to files using the openssl api, and verified them as good -- both the old stolen e/n and new ones I generated. Jan 13, 2023 at 17:53

1 Answer 1


Turns out if you're gonna get hex strings printed to console, you have to convert them back to a binary char array before decrypting..

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.