RSA_public_decrypt and equivalent MS Crypto API

Question

I am trying to develop a solution for license verification. Licenses are encoded on the server using the OpenSSL RSA_private_encrypt function.

For Mac OX X, I use RSA_public_decrypt and it works like a charm. On Windows, I have to use a very small bit of code, so I can’t communicate with OpenSSL or another library. And I have to use the MS Crypto API.

I spent several days trying to figure out what was wrong, but no luck. I can successfully import the public key, but here my success ends. I know that I need to change the byte order using CAPI, so this may not be a problem.

I tried everything including CryptVerifyMessageSignatureWithKey and CryptDecodeObject to load blob with different parameters, but still no luck.

It always ends with GetLastError() == CRYPT_E_ASN1_BADTAG , which I believe means that the BLOB is not formatted in ASN1 ... Google does not report anything about the output format RSA_private_encrypt ... so I got completely lost here.

Here is the OS X code based on OpenSSL:

 void cr_license_init(const char* lic) { __cr_license_ = lic; unsigned char lic_encoded[CR_LIC_LEN]; BIO* b64 = BIO_new(BIO_f_base64()); BIO* licIn = BIO_new_mem_buf((void*)lic, -1); licIn = BIO_push(b64, licIn); if(BIO_read(licIn, lic_encoded, CR_LIC_LEN) == CR_LIC_LEN) { const unsigned char* key_data = license_pub_der; RSA* r = d2i_RSA_PUBKEY(NULL, &key_data, sizeof(license_pub_der)); if(r != NULL) { if(__cr_license_data_ != NULL) { free((void*)__cr_license_data_); } __cr_license_data_ = malloc(CR_LIC_LEN); if(RSA_public_decrypt(CR_LIC_LEN, lic_encoded, (unsigned char*)__cr_license_data_, r, RSA_PKCS1_PADDING) &lt= 0) { free((void*)__cr_license_data_); __cr_license_data_ = NULL; } RSA_free(r); } } BIO_free_all(licIn); } 

This piece of code on windows works well, so I assume the public key is not a problem.

 __cr_license_ = lic; unsigned char lic_encoded[CR_LIC_LEN]; DWORD dwSize; if(CryptStringToBinaryA(__cr_license_, 0/*autocalculate*/, CRYPT_STRING_BASE64, lic_encoded, &dwSize, NULL, NULL) && dwSize == CR_LIC_LEN) { HCRYPTPROV hProv; if(CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) { PCERT_PUBLIC_KEY_INFO pki = NULL; DWORD dwKeySize; if(CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, license_pub_der, sizeof(license_pub_der), CRYPT_ENCODE_ALLOC_FLAG, NULL, &pki, &dwKeySize)) { HCRYPTKEY hKey = 0; if(CryptImportPublicKeyInfo( hProv, X509_ASN_ENCODING, pki, &hKey)) { 

But after that, everything I try to do with the message results in CRYPT_E_ASN1_BADTAG . I tried CryptMsgOpenToDecode with CryptMsgUpdate , CryptDecodeObject , CryptVerifyMessageSignatureWithKey - nothing works.

Basically, I think the problem is the incompatibility of pkcs1 and pkcs7, as mentioned above. Does anyone have any experience with importing / converting pkcs1 / etc format with MS CAPI?

Any help or even a hint is greatly appreciated! Thanks in advance!

Answer 1

You mix higher and lower level signature formats. OpenSSL by default signs PKCS # 1 v1.5 signatures that contain only signature data. Windows seems to use PKCS # 7 containers. They may contain PKCS # 1 v1.5, but this and other data is wrapped using the ASN.1 BER tag / length format. If the Microsoft API attempts to decode this, it is assumed that the raw signature is a container format and decoding will fail.

Answer 2

If it is so obvious that you tried, but did not specify it, or I misunderstood your question, I think you should use CryptDecrypt to decrypt the license, not the functions that you mentioned in the question. Please note that since you seem to be using OpenSSL with the addition of PKCS # 1 v1.5, and CryptoAPI does not seem to support this (not tested, but only a list of only PKCS # 1 v2 OAEP), you will probably have to use CRYPT_DECRYPT_RSA_NO_PADDING_CHECK and check and remove the PKCS # 1 v1.5 add-on manually after decryption.

Answer 3

OpenSSL exports keys with an optional header that is not expected by CryptoAPI.

Header for the private key (in ASN.1 notation):

 Offset| Len |LenByte| ======+======+=======+====================================================================== 0| 630| 3| SEQUENCE : 4| 1| 1| INTEGER : 0 7| 13| 1| SEQUENCE : 9| 9| 1| OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1] 20| 0| 1| NULL : 22| 608| 3| OCTET STRING : ... actual key data go here ... 

Header for the public key (in ASN.1 notation):

 Offset| Len |LenByte| ======+======+=======+====================================================================== 0| 159| 2| SEQUENCE : 3| 13| 1| SEQUENCE : 5| 9| 1| OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1] 16| 0| 1| NULL : 18| 141| 2| BIT STRING UnusedBits:0 : ... actual key data go here ... 

These headers are what make CryptDecodeObjectEx suffocate. It expects RAW key data without a header.

So basically you need:

• (Optional) Convert .PEM to .DER with CryptStringToBinary.
• Check if DER starts with the above headers. To do this, you need to read the data encoded by ASN.1.
• (Optional) Skip the above header and directly access the key data (starts with SEQUENCE, which includes 2 INTEGER for the public key or 9 INTEGER for the private key).
• Submit the result to CryptDecodeObjectEx (X509_ASN_ENCODING, RSA_CSP_PUBLICKEYBLOB / PKCS_RSA_PRIVATE_KEY).
• Import keys using CryptImportKey.