Have Geek Brain

How do you know what special symbol is in a file?

My application needs to process text files during batch process. Sometimes I get a file with some special character at the end of the file. I'm not sure what this special character is. Anyway, I can find what this symbol is so that I can tell the other command that produces this file.

I used the mozilla library to guess the encoding of a file and it says UTF-8.

+2


a source to share


3 answers


First, if the character is truly "special" or not dependent on what you call "special character". As a side-item on Unix and OS X, you can, for example, use the od, file, and hexdump commands to easily examine files:

... $  hexdump -C example.txt 
00000530  6f 77 73 20 61 63 74 69  6f 6e 2e 0a 0a 0a 0a     |ows action.....|

Now, if you know that your file encoding is UTF-8, this means that each byte with the highest bit set to zero matches exactly one character (in the example above, the last byte is "0a", 0a 'matches one character.

A UTF-8 file also means that every byte with the most significant bit of 1 is part of a multibyte character. For example, in the following sequence of bytes:

75 20 5b e2 80 a6 5d 20  61 75 74 6f 72 69 73 61


the only three bytes that have their most significant bit set are "e2 80 a6" (all values ​​from 0x80 to 0xFF have their most significant / most significant bit) and they are part of the same character (you cannot have -ASCII- a character in UTF-8, consisting of only one byte with the most significant bit set, so you know these three bytes are part of the same character ... The fact that every UTF-8 byte that has the most significant / the most significant bit is set, IMHO is a truly beautiful UTF-8 feature).

Now you Google on "e2 80 a6" and you can see that there is a Unicode character named "horizontal ellipsis" (whose UTF-8 code is represented by hex e280a6).

So basically you need to do two things:

  • find which bytes make up the last "special" character (is it just one byte or multiple bytes?)

  • find which "special character" this / these bytes correspond to

+4


a source


Any hex editor should allow you to see every single byte in the file. This should allow you to tell them what the symbol is.



Here's what I've used in the past: http://www.hexworkshop.com/

+1


a source


On Unix, you can use a utility od

to output multiple representations of byte data to a file or stream.

+1


a source






More articles:


All Articles