Is there any way to understand this, other than reverse engineering the method itself?
PKCS5_PBKDF2_HMAC_SHA1 looks like one of those undocumented functions because I cannot find it in OpenSSL docs . OpenSSL has a lot of them, so you should be prepared to study the sources if you intend to use the library.
I understand that it returns 0 if it succeeds, and another otherwise.
Actually, its the opposite. This is how I know ...
$ grep -R PKCS5_PBKDF2_HMAC_SHA1 * crypto/evp/evp.h:int PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen, crypto/evp/p5_crpt2.c:int PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen, ...
So, you will find the implementation of the function in crypto/evp/p5_crpt2.c :
int PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen, const unsigned char *salt, int saltlen, int iter, int keylen, unsigned char *out) { return PKCS5_PBKDF2_HMAC(pass, passlen, salt, saltlen, iter, EVP_sha1(), keylen, out); }
Following PKCS5_PBKDF2_HMAC :
$ grep -R PKCS5_PBKDF2_HMAC * ... crypto/evp/evp.h:int PKCS5_PBKDF2_HMAC(const char *pass, int passlen, crypto/evp/p5_crpt2.c:int PKCS5_PBKDF2_HMAC(const char *pass, int passlen, ...
And again, from crypto/evp/p5_crpt2.c :
int PKCS5_PBKDF2_HMAC(const char *pass, int passlen, const unsigned char *salt, int saltlen, int iter, const EVP_MD *digest, int keylen, unsigned char *out) { unsigned char digtmp[EVP_MAX_MD_SIZE], *p, itmp[4]; int cplen, j, k, tkeylen, mdlen; unsigned long i = 1; HMAC_CTX hctx_tpl, hctx; mdlen = EVP_MD_size(digest); if (mdlen < 0) return 0; HMAC_CTX_init(&hctx_tpl); p = out; tkeylen = keylen; if(!pass) passlen = 0; else if(passlen == -1) passlen = strlen(pass); if (!HMAC_Init_ex(&hctx_tpl, pass, passlen, digest, NULL)) { HMAC_CTX_cleanup(&hctx_tpl); return 0; } while(tkeylen) { if(tkeylen > mdlen) cplen = mdlen; else cplen = tkeylen; itmp[0] = (unsigned char)((i >> 24) & 0xff); itmp[1] = (unsigned char)((i >> 16) & 0xff); itmp[2] = (unsigned char)((i >> 8) & 0xff); itmp[3] = (unsigned char)(i & 0xff); if (!HMAC_CTX_copy(&hctx, &hctx_tpl)) { HMAC_CTX_cleanup(&hctx_tpl); return 0; } if (!HMAC_Update(&hctx, salt, saltlen) || !HMAC_Update(&hctx, itmp, 4) || !HMAC_Final(&hctx, digtmp, NULL)) { HMAC_CTX_cleanup(&hctx_tpl); HMAC_CTX_cleanup(&hctx); return 0; } HMAC_CTX_cleanup(&hctx); memcpy(p, digtmp, cplen); for(j = 1; j < iter; j++) { if (!HMAC_CTX_copy(&hctx, &hctx_tpl)) { HMAC_CTX_cleanup(&hctx_tpl); return 0; } if (!HMAC_Update(&hctx, digtmp, mdlen) || !HMAC_Final(&hctx, digtmp, NULL)) { HMAC_CTX_cleanup(&hctx_tpl); HMAC_CTX_cleanup(&hctx); return 0; } HMAC_CTX_cleanup(&hctx); for(k = 0; k < cplen; k++) p[k] ^= digtmp[k]; } tkeylen-= cplen; i++; p+= cplen; } HMAC_CTX_cleanup(&hctx_tpl); return 1; }
Thus, it looks like 0 on failure, and 1 on success. You should not see other values. And if you get 0 , then all OUT parameters are undesirable.
Memory error Usage error?
Well, sometimes you can call ERR_get_error . If you call it, and it makes sense, then the error code is good. If the error code does not make sense, then this is probably not very good.
Unfortunately, this is how I deal with this because the library is not consistent with setting error codes. For example, here is the library code for loading the RDRAND engine.
Please note that the code clears the error code if it fails if its third-generation Ivy Bridge (a tested feature) does not clear or does not set the error otherwise.
void ENGINE_load_rdrand (void) { extern unsigned int OPENSSL_ia32cap_P[]; if (OPENSSL_ia32cap_P[1] & (1<<(62-32))) { ENGINE *toadd = ENGINE_rdrand(); if(!toadd) return; ENGINE_add(toadd); ENGINE_free(toadd); ERR_clear_error(); } }
How should my program handle (repeat, exit?)?
Sounds like a hard failure.
Finally, this is how I navigate the sources in this situation. If you don't like grep , you can try ctags or another source code browser.