Decoding does not override Unicode encoding in Django / Python
Ok, I have a hardcoded string which I declare as
name = u"Par Catégorie"
I have # -encoding: utf-8 is the magic header, so I assume it is converted to utf-8
Down the road he sent to xml via
xml_output.toprettyxml(indent='....', encoding='utf-8')
And i get
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
Most of my data is written in French and printed correctly on CDATA nodes, but this one line is anchored ... I don't understand why the ascii codec is being called.
what's wrong?
source to share
The header coding
in the source file tells Python what encoding your source is in. Its Python encoding uses to decode the source of a unicode string literal ( u"Par Catégorie"
) into a unicode object. The unicode object itself is not encoded; it is raw Unicode data. (Internally, Python will use one of two encodings, depending on how it was configured, but Python code shouldn't worry about that.)
The UnicodeDecodeError you are getting means that somewhere you are mixing unicode and bytestrings (regular strings). By mixing the two (concatenating, doing string interpolation, etc.), Python will try to convert the bytestring to a unicode string by decoding the bytestring using the default encoding, ASCII. If the byte string contains non-ASCII data, it will crash with the error you are seeing. The operation being performed may be in the library somewhere, but that still means you are mixing inputs of different types.
Unfortunately, the fact that it will work fine as long as the bytes only contain ASCII data means that this type of error is all too common, even in library code. Python 3.x solves this problem by getting rid of the implicit conversion of unicode strings (just str
in 3.x) and bytestrings (type bytes
in 3.x.)
source to share