Error connecting to the azure blob storage API from R

Question

I am trying to work with the Azure repository through the REST API in R. I am using the httr package that Curl imposes.

Customization

You can use the R script: http://www.r-fiddle.org/#/fiddle?id=vh8uqGmM

 library(httr) requestdate<-format(Sys.time(),"%a, %d %b %Y %H:%M:%S GMT") url<-"https://preconstuff.blob.core.windows.net/pings?restype=container&comp=list" sak<-"Q8HvUVJLBJK+wkrIEG6LlsfFo19iDjneTwJxX/KXSnUCtTjgyyhYnH/5azeqa1bluGD94EcPcSRyBy2W2A/fHQ==" signaturestring<-paste0("GET",paste(rep("\n",12),collapse=""), "x-ms-date:",requestdate," x-ms-version:2009-09-19 /preconstuff/pings comp:list restype:container") headerstuff<-add_headers(Authorization=paste0("SharedKey preconstuff:", RCurl::base64(digest::hmac(key=sak, object=enc2utf8(signaturestring), algo= "sha256"))), `x-ms-date`=requestdate, `x-ms-version`= "2009-09-19") 

Trying to list blobs:

 content(GET(url,config = headerstuff, verbose() )) 

Error

Top Level Message

The MAC signature found in the HTTP request 'Q8HvUVJLBJK + wkrIEG6LlsfFo19iDjneTwJxX / KXSnUCtTjgyyhYnH / 5azeqa1bluGD94EcPcSRyBy2W2A / fHQ ==' does not match any calculation.

Response content

 [1] "<?xml version=\"1.0\" encoding=\"utf-8\"?><Error> <Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:1ab26da5-0001-00dc-6ddb-15e35c000000\nTime:2015-03-26T17:51:42.7190620Z</Message> <AuthenticationErrorDetail>The MAC signature found in the HTTP request 'NTM1ODZjMjhhZmMyZGM3NDM0YTFjZDgwNGE0ODVmMzVjNDhkNjBkNzk1ZjNkZjJjOTNlNjUxYTMwMjRhNzNlYw==' is not the same as any computed signature. Server used following string to sign: 'GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 26 Mar 2015 17:52:37 GMT\nx-ms-version:2009-09-19\n/preconstuff/pings\ncomp:list\nrestype:container'. </AuthenticationErrorDetail></Error>" 

Verbose conclusion

 -> GET /pings?restype=container&comp=list HTTP/1.1 -> User-Agent: curl/7.39.0 Rcurl/1.95.4.5 httr/0.6.1 -> Host: preconstuff.blob.core.windows.net -> Accept-Encoding: gzip -> Accept: application/json, text/xml, application/xml, */* -> Authorization: SharedKey preconstuff:OTRhNTgzYmY3OTY3M2UzNjk3ODdjMzk3OWM3ZmU0OTA4MWU5NTE2OGYyZGU3YzRjNjQ1M2NkNzY0ZTcyZDRhYQ== -> x-ms-date: Thu, 26 Mar 2015 17:56:27 GMT -> x-ms-version: 2009-09-19 -> <- HTTP/1.1 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. <- Content-Length: 719 <- Content-Type: application/xml <- Server: Microsoft-HTTPAPI/2.0 <- x-ms-request-id: 3d47770c-0001-0085-2313-6d466f000000 <- Date: Thu, 26 Mar 2015 17:56:27 GMT <- 

Error resolution

Googling for this problem does not seem to provide a consistent reason, but probably due to the poor formatting / query structure on my part. For this, I checked:

• I checked that my key is correct (this is just c & p from the portal)
• I am sure the date is formatted correctly.
• There was a recent documentDB SO which suggested that this could be a clock skew problem, and I note that my x-ms-date is one second ahead of the Date in the response. I tried to send a fixed value that was definitely in the past, but for 15 minutes. Did not receive a change in the message.
• Added encoding="Base64" in the headerstuff later in the MSDN forum question , but the same error message was sent
• In response to @Serdar, I included the signature string construction (I checked that it matches the one indicated in the error message, therefore), then base64 hmac-sha256 encoding (using the secondary access key ( sak ) as the encryption key) UTF8 version converts signaturestring as the value to be used in SharedKey authorization.
• In addition to the comment by @Serdar, the date used in the signature line and for the main request must be the same, therefore it is determined once and reused

Is there something clearly wrong? Is there anything else to check? Does the code work for others?

Answer 1

It looks like your problem is with the key. The key string you provided is actually base64 encoded. You must decode this to a raw vector before using it to sign a request. For instance:

 url<-"https://preconstuff.blob.core.windows.net/pings?restype=container&comp=list" sak<-"Q8HvUVJLBJK+wkrIEG6LlsfFo19iDjneTwJxX/KXSnUCtTjgyyhYnH/5azeqa1bluGD94EcPcSRyBy2W2A/fHQ==" requestdate<-format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT") signaturestring<-paste0("GET",paste(rep("\n",12),collapse=""), "x-ms-date:",requestdate," x-ms-version:2009-09-19 /preconstuff/pings comp:list restype:container") headerstuff<-add_headers(Authorization=paste0("SharedKey preconstuff:", RCurl::base64(digest::hmac(key=RCurl::base64Decode(sak, mode="raw"), object=enc2utf8(signaturestring), algo= "sha256", raw=TRUE))), `x-ms-date`=requestdate, `x-ms-version`= "2009-09-19") content(GET(url,config = headerstuff, verbose() )) 

In this case, there are no more authorization errors, although there are no drops in the list. Perhaps this is another problem.

In addition, I changed the way the date / time is created to “safely” change the local time to GMT.

Answer 2

It looks like you are using your account key directly in the authorization header. In order to authenticate the request, you must sign the request with the key for the account that makes the request, and pass this signature as part of the request. For more information on how to create an authorization header, see Authentication for Azure Storage Services .

Also note that the service returns StringToSign in the error response. So what your code should have done is apply the following formula to StringToSign = "GET \ n \ n \ n \ n \ n \ n \ n \ n \ n \ n \ n \ nx-ms-date: Wed, 25 March 2015 22:24:12 GMT \ nx-ms version: 2014-02-14 \ n / preconstuff / pings \ ncomp: list \ nrestype: container "(without quotes):

 Signature=Base64(HMAC-SHA256(AccountKey, UTF8(StringToSign))) 

How the service computes StringToSign is explained in detail in the general link above.

Answer 3

While working on the MrFlick sample code above, and to get it working, I had to change a few things.

The date string must be set in English, for example:

 lct <- Sys.getlocale("LC_TIME") Sys.setlocale("LC_TIME", "us") requestdate <- format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT") Sys.setlocale("LC_TIME", lct) 

"signaturestring" must be formed with \ n between the parameters:

  signaturestring <- paste0("GET", paste(rep("\n", 12), collapse=""), "x-ms-date:", requestdate, "\nx-ms-version:2009-09-19\n/preconstuff/pings\ncomp:list\nrestype:container") 

EDIT: The following procedure works for me. Based on the example of Stef Locke.

 library(httr) library(RCurl) azureBlobCall <- function(url, verb, key, requestBody=NULL, headers=NULL, ifMatch="", md5="") { urlcomponents <- httr::parse_url(url) account <- gsub(".blob.core.windows.net", "", urlcomponents$hostname, fixed = TRUE) container <- urlcomponents$path # get timestamp in us locale lct <- Sys.getlocale("LC_TIME"); Sys.setlocale("LC_TIME", "us") `x-ms-date` <- format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT") Sys.setlocale("LC_TIME", lct) # if requestBody exist get content length in bytes and content type `Content-Length` <- ""; `Content-Type` <- "" if(!is.null(requestBody)) { if(class(requestBody) == "form_file") { `Content-Length` <- (file.info(requestBody$path))$size `Content-Type` <- requestBody$type } else { requestBody <- enc2utf8(as.character(requestBody)) `Content-Length` <- nchar(requestBody, "bytes") `Content-Type` <- "text/plain; charset=UTF-8" } } # combine timestamp and version headers with any input headers, order and create the CanonicalizedHeaders headers <- setNames(c(`x-ms-date`, "2015-04-05", unlist(headers)), c("x-ms-date", "x-ms-version", unclass(names(unlist(headers))))) headers <- headers[order(names(headers))] CanonicalizedHeaders <- paste(names(headers), headers, sep=":", collapse = "\n") # create CanonicalizedResource headers and add any queries to it if(!is.null(urlcomponents$query)) { components <- setNames(unlist(urlcomponents$query), unclass(names(unlist(urlcomponents$query)))) componentstring <- paste0("\n", paste(names(components[order(names(components))]), components[order(names(components))], sep=":", collapse = "\n")) } else componentstring <- "" CanonicalizedResource <- paste0("/",account,"/",container, componentstring) # create the authorizationtoken signaturestring <- paste0(verb, "\n\n\n", `Content-Length`, "\n", md5, "\n", `Content-Type`, "\n\n\n", ifMatch, "\n\n\n\n", CanonicalizedHeaders, "\n", CanonicalizedResource) requestspecificencodedkey <- RCurl::base64( digest::hmac(key=RCurl::base64Decode(key, mode="raw"), object=enc2utf8(signaturestring), algo= "sha256", raw=TRUE) ) authorizationtoken <- paste0("SharedKey ", account, ":", requestspecificencodedkey) # make the call headers_final <- add_headers(Authorization=authorizationtoken, headers, `Content-Type` = `Content-Type`) call <- httr::VERB(verb=verb, url=url, config=headers_final, body=requestBody, verbose()) print("signaturestring");print(signaturestring); print(headers_final); print(call) return(content(call)) } ## Tests. Replace 'key' and 'accountName' with yours key <- "YowThr***********RDw==" # Creates a container named 'test' azureBlobCall("https://accountName.blob.core.windows.net/test?restype=container", "PUT", key) # Creates a blob named 'blob' under container 'test' with the content of "Hej världen!" azureBlobCall("https://accountName.blob.core.windows.net/test/blob", "PUT", key, headers = c("x-ms-blob-type"="BlockBlob"), requestBody = "Hej världen!") #upload_file("blob.txt")) # List all blob in the container 'test' azureBlobCall("https://accountName.blob.core.windows.net/test?comp=list&restype=container", "GET", key) # deletes the blobl named 'blob' azureBlobCall("https://accountName.blob.core.windows.net/test/blob", "DELETE", key) # Creates a blob named 'blob' under container 'test' with and upload the file 'blob.txt' azureBlobCall("https://accountName.blob.core.windows.net/test/blob", "PUT", key, headers = c("x-ms-blob-type"="BlockBlob"), requestBody = upload_file("blob.txt")) # deletes the container named 'test' azureBlobCall("https://accountName.blob.core.windows.net/test?restype=container", "DELETE", key)