Convert Python dictionary to JSON array

Question

I currently have this dictionary printed using pprint :

 {'AlarmExTempHum': '\x00\x00\x00\x00\x00\x00\x00\x00', 'AlarmIn': 0, 'AlarmOut': '\x00\x00', 'AlarmRain': 0, 'AlarmSoilLeaf': '\x00\x00\x00\x00', 'BarTrend': 60, 'BatteryStatus': 0, 'BatteryVolts': 4.751953125, 'CRC': 55003, 'EOL': '\n\r', 'ETDay': 0, 'ETMonth': 0, 'ETYear': 0, 'ExtraHum1': None, 'ExtraHum2': None, 'ExtraHum3': None, 'ExtraHum4': None, 'ExtraHum5': None, 'ExtraHum6': None, 'ExtraHum7': None, 'ExtraTemp1': None, 'ExtraTemp2': None, 'ExtraTemp3': None, 'ExtraTemp4': None, 'ExtraTemp5': None, 'ExtraTemp6': None, 'ExtraTemp7': None, 'ForecastIcon': 2, 'ForecastRuleNo': 122, 'HumIn': 31, 'HumOut': 94, 'LOO': 'LOO', 'LeafTemps': '\xff\xff\xff\xff', 'LeafWetness': '\xff\xff\xff\x00', 'NextRec': 37, 'PacketType': 0, 'Pressure': 995.9363359295631, 'RainDay': 0.0, 'RainMonth': 0.0, 'RainRate': 0.0, 'RainStorm': 0.0, 'RainYear': 2.8, 'SoilMoist': '\xff\xff\xff\xff', 'SoilTemps': '\xff\xff\xff\xff', 'SolarRad': None, 'StormStartDate': '2127-15-31', 'SunRise': 849, 'SunSet': 1611, 'TempIn': 21.38888888888889, 'TempOut': 0.8888888888888897, 'UV': None, 'WindDir': 219, 'WindSpeed': 3.6, 'WindSpeed10Min': 3.6} 

When I do this:

 import json d = (my dictionary above) jsonarray = json.dumps(d) 

I get this error: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte

Answer 1

If everything is fine with non-printable characters in json, add a call to ensure_ascii=False to dumps .

 >>> json.dumps(your_data, ensure_ascii=False) 

If ensure_ascii is false, then the return value will be a unicode instance with normal Python str until the unicode enforcement rule instead of escaping to ASCII str .

Answer 2

secure_ascii = False really only discards the problem at the decoding stage:

 >>> dict2 = {'LeafTemps': '\xff\xff\xff\xff',} >>> json1 = json.dumps(dict2, ensure_ascii=False) >>> print(json1) {"LeafTemps": "    "} >>> json.loads(json1) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/json/__init__.py", line 328, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 365, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 381, in raw_decode obj, end = self.scan_once(s, idx) UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte 

Ultimately, you cannot store raw bytes in a JSON document, so you might want to use some way to uniquely encode a sequence of arbitrary bytes as ASCII strings, such as base64.

 >>> import json >>> from base64 import b64encode, b64decode >>> my_dict = {'LeafTemps': '\xff\xff\xff\xff',} >>> my_dict['LeafTemps'] = b64encode(my_dict['LeafTemps']) >>> json.dumps(my_dict) '{"LeafTemps": "/////w=="}' >>> json.loads(json.dumps(my_dict)) {u'LeafTemps': u'/////w=='} >>> new_dict = json.loads(json.dumps(my_dict)) >>> new_dict['LeafTemps'] = b64decode(new_dict['LeafTemps']) >>> print new_dict {u'LeafTemps': '\xff\xff\xff\xff'} 

Answer 3

If you are using Python 2, be sure to add a comment on the encoding of the UTF-8 file in the first line of your script.

 # -*- coding: UTF-8 -*- 

This will fix some Unicode issues and make your life easier.

Answer 4

One possible solution I'm using is using python3. It seems to solve many problems with utf.

Sorry for the late reply, but this may help people in the future.

For example,

 #!/usr/bin/env python3 import json # your code follows