Twitter trends api UnicodeDecodeError: codec 'utf8' cannot decode byte 0x8b at position 1: unexpected code byte

Question

I am trying to follow the exemplary code of the book "Extraction of a social network", 1-3.

I know it is old, so I follow the new sample from the web page enter the link description here

BUT, SOMETIMES, I will experience an error while implementing the code:

[ trend.decode('utf-8') for trend in world_trends()[0]['trends'] ] 

And the error information looks like this:

 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "build/bdist.macosx-10.6-universal/egg/twitter/api.py", line 167, in __call__ File "build/bdist.macosx-10.6-universal/egg/twitter/api.py", line 173, in _handle_response File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte 

This does not always happen, but I think that no programmer likes this random case.

So can anyone help me on this issue? What is the problem and how can I solve it?

Thanks a lot ~

Answer 1

byte 0x8b in position 1 usually signals that the data stream is gzipped. For similar problems see here and here .

To unzip a data stream:

 buf = StringIO.StringIO(<response object>.content) gzip_f = gzip.GzipFile(fileobj=buf) content = gzip_f.read() 

Answer 2

By default, decode () throws an error if it encounters a byte that it does not know how to decode.

You can use trend.decode('utf-8', 'replace') or trend.decode('utf-8', 'ignore') to not throw an error and silently ignore it.

Decoding documentation () here.