First of all, it is quite unlikely that the key has a passphrase. Passphrases are used to defend against stolen key assets, but since the key asset is embedded inside the program together with any passphrase, this is redundant. Therefore I will not assume there's a passphrase.
Now, to your main question. libeay32.dll exports many encryption related functions, and those are presumably used by said program. Without any more specifics, I'd suggest you search for the functions imported into the program from libeay32.dll.
Depending on the algorithm used (my first bet would be AES) to decrypt the files, several different functions will be called. The one you're interested in, that loads / initializes a key, a common such API is EVP_CIPHER_CTX_new which allocates a new symmetric encryption/decryption context, which will then be used in a function such as EVP_DecryptInit or EVP_DecryptInit_ex which will actually accept the key and iv parameters.
Without reviewing the APIs imported, it is quite hard to list the possible key-initialization functions you may be looking for. For example, EVP_PKEY_decrypt_init will be an equivalent function used for public key decryption (which could also be used in your case, if for some reason author decided to use asymmetric cryptography).
Once you identify the key initialization function, this should be a matter of tracking the values put into the key and iv parameters. A hardcoded string will be easy to find while a more complex key-generation flow (maybe using a hash function?) could be used. In the case of file decryption, it is unlikely that the key is not deterministic so it should be easy enough.
If you wish to manually decrypt the files, there are other important parameters except the key and iv. The encryption algorithm, mode of operation, block size, etc...
