Best way to parse ANSI and UTF-16LE files using Python 2/3?
I have a collection of files encoded in ANSI or UTF-16LE. I would like python to open files using the correct encoding. The problem is that ANSI files don't throw any kind of exception when encoded using UTF-16le and vice versa.
Is there an easy way to open files using the correct file encoding?
source to share
You can check the BOM at the beginning of the file to check if it is UTF.
Then unicode.decode accordingly (using one of the standard encodings ).
EDIT Or maybe try s.decode ('ascii') your string (the given s is the name of the variable). If it throws a UnicodeDecodeError, it decodes it as "utf_16_le".
source to share
What's in the files? If it is plain text in the Latin alphabet, almost every other byte, UTF-16LE files will be zero. On the other hand, I didn't expect to see zeros in windows-1252 files. For example, here "Hello"
on windows-1252:
93 48 65 6C 6C 6F 94
... and in UTF-16LE:
1C 20 48 00 65 00 6C 00 6C 00 6F 00 1D 20
In addition to curly quotes, each character is matched against the same value with a trailing null byte appended. In fact, this is true for every character in the ISO-8859-1 character set (windows-1252 extends ISO-8859-1 to add collations for multiple printable characters, such as curly quotes, to replace control characters in a range 0x80..0x9F
).
If you know that all files are either windows-1252 or UTF-16LE, you need to quickly scan over the zeros to figure out what exactly. There's a good reason why the chart is so slow and complex, but in this case I think you can get away quickly and dirty.
source to share