MD5-SHA1 using CryptoAPI

Question

I have a requirement to sign data using the MD5-SHA1 hash (the MD5 hash and the SHA1 hash of the data are combined and then signed).

Primary requirement:

The MD5-SHA1 hash is provided by OpenSSL in some way that is not displayed. This hash is complete.

Now my requirement is to sign this hash using the Crypto API without hashing (only signing is required, not hashing). Why is CryptoAPI and not OpenSSL? Because I am dealing with a certificate with a non-exportable private key. Its private key can only be used by CryptoAPI, not OpenSSL.

This scenario is connected with SSL communication, where the client part of the verification is sent (in the case of a non-exported private key) to my level and tries to sign the data using CryptoAPI.

In addition, I would like to learn about other ways to use non-exported private key certificates using OpenSSL.

Answer 1

Using CALG_SSL3_SHAMD5 in CryptCreateHash and then calling CryptSetHashParam and then calling CryptSignHash did the job for me.

Let me clarify the requirement: 1. As a rule, I provided the private key, using EVP_PKEY, for the OpenSSL function so that it could execute part of the signature. I used to extract the secret key from the Windows certificate store (when the key is marked as exportable). But in one scenario, the private key is not marked as exportable, and I could not get the private key for EVP_PKEY. Thus, I made a modification of the OpenSSL code in the case when the private key is not available, then part of the signature is implemented using CryptoAPI (since CryptoAPI allows signing using such a non-exported key).

So, in my case, the data is completely hashed by OpenSSL (using the md5-sha1 hash). The only thing left for me is to subscribe.

So, the following steps (signed) completed the task:

1. Created hash using Certificate key handle provider and use CALG_SSL3_SHAMD5 algorithm. 2. Using CryptSetHashParam, I set the initial hash value 3. Signed the hash using CryptSignHash. 4. After that, reverse the bytes order (as OpenSSL signature is Big endian mentioned [here][1] which I found after lots of experimenting). 

The rest of the OpenSSL code seems pleased with this signature.