Got the error "invalid character", "looking for the beginning of the value" from json.Unmarshal

Question

I am using the Golang HTTP request to get json output as follows. The web service I'm trying to access is Micrsoft Translator https://msdn.microsoft.com/en-us/library/dn876735.aspx

//Data struct of TransformTextResponse type TransformTextResponse struct { ErrorCondition int `json:"ec"` // A positive number representing an error condition ErrorDescriptive string `json:"em"` // A descriptive error message Sentence string `json:"sentence"` // transformed text } //some code .... body, err := ioutil.ReadAll(response.Body) defer response.Body.Close() if err != nil { return "", tracerr.Wrap(err) } transTransform = TransformTextResponse{} err = json.Unmarshal(body, &transTransform) if err != nil { return "", tracerr.Wrap(err) } 

I received an error message invalid character 'ï' looking for beginning of value

So, I'm trying to print the string body as the string fmt.Println(string(body)) , it shows:

 {"ec":0,"em":"OK","sentence":"This is too strange i just want to go home soon"} 

The data seems to have no problems, so I tried to create the same jason.Marshal value

 transTransform := TransformTextResponse{} transTransform.ErrorCondition = 0 transTransform.ErrorDescriptive = "OK" transTransform.Sentence = "This is too strange i just want to go home soon" jbody, _ := json.Marshal(transTransform) 

I found that the source data may have problems, so I am trying to compare two data in the []byte format.

Data from response.Body :

 [239 187 191 123 34 101 99 34 58 48 44 34 101 109 34 58 34 79 75 34 44 34 115 101 110 116 101 110 99 101 34 58 34 84 104 105 115 32 105 115 32 116 111 111 32 115 116 114 97 110 103 101 32 105 32 106 117 115 116 32 119 97 110 116 32 116 111 32 103 111 32 104 111 109 101 32 115 111 111 110 34 125] 

Data from json.Marshal

 [123 34 101 99 34 58 48 44 34 101 109 34 58 34 79 75 34 44 34 115 101 110 116 101 110 99 101 34 58 34 84 104 105 115 32 105 115 32 116 111 111 32 115 116 114 97 110 103 101 32 105 32 106 117 115 116 32 119 97 110 116 32 116 111 32 103 111 32 104 111 109 101 32 115 111 111 110 34 125] 

Any idea how I parse this response.Body and decouple it into a data structure?

Answer 1

The server sends you a UTF-8 text string with a Byte Byte (BOM) mark . The specification indicates that the text is encoded in UTF-8 encoding, but it must be deleted before decoding.

This can be done with the following line (using package "bytes" ):

 body = bytes.TrimPrefix(body, []byte("\xef\xbb\xbf")) // Or []byte{239, 187, 191} 

PS. The error related to ï is that the UTF-8 specification, interpreted as the string ISO-8859-1, will result in  characters.