Encoding a response using the node.js "request" module

Question

I am trying to get data from the Bing search APIs, and since existing libraries seem to be based on older APIs that were discontinued, I would try using the request library, which seems to be the most common library for this. My code looks like

 var SKEY = "myKey...." , ServiceRootURL = 'https://api.datamarket.azure.com/Bing/Search/v1/Composite'; function getBingData(query, top, skip, cb) { var params = { Sources: "'web'", Query: "'"+query+"'", '$format': "JSON", '$top': top, '$skip': skip }, req = request.get(ServiceRootURL).auth(SKEY, SKEY, false).qs(params); request(req, cb) } getBingData("bookline.hu", 50, 0, someCallbackWhichParsesTheBody) 

Bing returns some JSON, and sometimes I can work with it , but if the response body contains a large number of non-ASCII characters JSON.parse , it states that the string is incorrect. I tried switching to the ATOM content type, but there was no difference, the xml was invalid. Checking the response body, available in the request() callback, actually shows bad code.

So, I tried the same query with some Python code, and this works fine all the time. For reference:

 r = requests.get( 'https://api.datamarket.azure.com/Bing/Search/v1/Composite?Sources=%27web%27&Query=%27sexy%20cosplay%20girls%27&$format=json', auth=HTTPBasicAuth(SKEY,SKEY)) stuffWithResponse(r.json()) 

I can not reproduce the problem with smaller answers (for example, limiting the number of results) and could not identify the one result that causes the problem (by increasing the offset). My impression is that the answer is read in chunks, somehow transcoded and reassembled, which means that json / atom data becomes invalid if some multibyte character becomes split, which happens with large answers, but not in small ones.

Being a newbie to node, I'm not sure if I have to do anything (setting the encoding somewhere? Bing returns UTF-8, so this doesn't seem necessary).

Does anyone have an idea of ​​what is going on?

FWIW, I'm on OSX 10.8, node is v0.8.20 installed via macports, request v2.14.0 is installed via npm.

Answer 1

I'm not sure about the query library, but by default nodejs works well for me. It also seems a lot easier to read than your library, and really goes back to pieces.

http://nodejs.org/api/http.html#http_http_request_options_callback or for https (e.g. your req) http://nodejs.org/api/https.html#https_https_request_options_callback (however, true)

There’s a bit of hint for the parameters: use URL parsing

 var url = require('url'); var params = '{}' var dataURL = url.parse(ServiceRootURL); var post_options = { hostname: dataURL.hostname, port: dataURL.port || 80, path: dataURL.path, method: 'GET', headers: { 'Content-Type': 'application/json; charset=utf-8', 'Content-Length': params.length } }; 

obviously, the parameters must be the data you want to send

Answer 2

I think your request authentication is incorrect. Authentication must be provided before request.get. See the documentation for requesting HTTP authentication . qs is an object that must be passed request parameters in the same way as url and auth. Also you use the same req for the second query. You should know that request.get returns a stream for the GET url. Your next request using req will be erroneous.

If you only need HTTPBasicAuth, this should work too

 //remove req = request.get and subsequent request request.get('http://some.server.com/', { 'auth': { 'user': 'username', 'pass': 'password', 'sendImmediately': false } },function (error, response, body) { }); 

The callback argument receives 3 arguments. The first is an error when applicable (usually from the http.Client option, not the http.ClientRequest object). The second object is http.ClientResponse. The third is the response body of a String or Buffer. The second object is the response flow. To use it, you must use the event data ',' end ',' error 'and' close '.

Be sure to use the arguments correctly.

Answer 3

You need to pass the {json: true} option to enable json analysis of the response