The decrypted openssl base64 string is not always decrypted

Question

I am trying to encrypt some plain text with a public key using RSA_public_encrypt (), this data will then be sent to a remote server for verification. I believe that encryption / decryption works for me, since the output of RSA_public_encrypt can be passed to RSA_private_decrypt and it works. The problem that I am facing now is that I need base64 to encode the data in order to send it over HTTP.

As a test (before sending it to the server), I encode the output RSA_public_encrypt () on base64, then decode it and pass it back to RSA_private_decrypt (). It seems like this has been working for a while and looks like this with an error:

error:0407A079:rsa routines:RSA_padding_check_PKCS1_OAEP:oaep decoding error 

When I use memcmp to compare the source data (pre-base64) with the output of my base64 decoding function, I get -1, although the content should match (in Visual Studio, viewing the memory contents as hex), I also checked the base64 encoding with various online tools, and they seem to be decoded to the expected value.

I double-checked that the input / output from the base64 / unbase64 functions is null-terminated, which seems to be not much different.

I came across this problem a couple of times, but I believe that this should be something with the base64 encoding / decoding process, because when it is not connected, everything works. If anyone has any advice on how this could happen, that would be appreciated.

Openssl Version: 1.0.1c

Platform: Windows / MSVC

Base64 Code:

 char *base64(const unsigned char *input, int length) { BIO *bmem, *b64; BUF_MEM *bptr; char *buff = NULL; b64 = BIO_new(BIO_f_base64()); BIO_set_flags(b64, BIO_FLAGS_BASE64_NO_NL); bmem = BIO_new(BIO_s_mem()); b64 = BIO_push(b64, bmem); BIO_write(b64, input, length); BIO_flush(b64); BIO_get_mem_ptr(b64, &bptr); buff = (char *)malloc(bptr->length+1); memcpy(buff, bptr->data, bptr->length); buff[bptr->length] = '\0'; BIO_free_all(b64); return buff; } 

Unbase64 Code:

 char *unbase64(unsigned char *input, int length) { BIO *b64, *bmem; char *buffer = (char *)malloc(length+1); memset(buffer, 0, length+1); b64 = BIO_new(BIO_f_base64()); BIO_set_flags(b64, BIO_FLAGS_BASE64_NO_NL); bmem = BIO_new_mem_buf(input, length); bmem = BIO_push(b64, bmem); BIO_read(bmem, buffer, length); buffer[length] = '\0'; BIO_free_all(bmem); return buffer; } 

An example of an encrypted "hello world":

 š:Œ¼JŒ"ÌïëŸÔè#¢Oo‚À– œê\çrú¿±a/8ƒòÌ¢Q\T¹]nío 

Base64 version (using the code above):

 G5qdOgWMvEqMIswZ7+uf1OgPI6JPb4LAlgmc6lzncvq/sWEvOIPyzByiUVwMjYFUuV0Vbu1v 

Thanks for any help!

Answer 1

Are you passing the correct size in the unbase64 function? This should be the size of the base64 returned buffer, not the size of the destination buffer, i.e. Using the main function of the example:

 int main(void) { unsigned char bufron[2000]; int i; char *chab; unsigned char *chac; for (i = 0; i < 2000; i++) { bufron[i] = i % 255; } chab = base64(bufron, 2000); printf("%s\n", chab); chac = unbase64(chab, strlen(chab)); for (i = 0; i < 2000; i++) { if (bufron[i] != chac[i]) { printf("Failed at %d\n", i); return (1); } } }