MFC: what would be a regex to check if a character is unicode or not?
I am trying to use the Windows API IsTextUnicode to check if the character input is Unicode or not, but it is kind of an error. I thought it would be better to use regex. However, I am new to creating regular expressions. What would be a regex to test if a character is unicode or not?
Thanks...
a source to share
Well, it depends on what you mean by Unicode. As the answers say so far, almost any character is "Unicode".
Windows abuses the term "Unicode" to refer to the UTF-16LE encoding used by the Win32 API. You can spot UTF-16 by looking for the byte order mark in front, bytes FF FE
for UTF-16LE (or FE FF
for UTF-16BE). It is possible to have UTF-16 text that is not marked with a BOM, but this is bad news as you can only detect it with pure guesswork.
Pure guesswork is what a function is IsTextUnicode
. He looks at the input bytes and, seeing how often common patterns appear, he guesses how likely the bytes are to represent UTF-16LE or UTF-16BE characters. Since each sequence of bytes is potentially a valid character encoding (*), you can imagine that this is not very predictable or reliable. And you would be right.
See Windows i18n guru Michael Kaplan for a description IsTextUnicode
and why it's probably not a good idea.
In general, you want a more predictable way of guessing what a set of bytes encodes. You can try:
- if it starts with
FE FF
, it is UTF-16LE, which is what Windows thinks of as "Unicode"; - if it starts with
FF FE
, it is UTF-16BE, which is what Windows calls inverse Unicode equivalently; - otherwise, check the entire string for invalid UTF-8 sequences. If not, perhaps UTF-8 (or just ASCII);
- otherwise try the default codepage.
(*: actually not quite true). Besides non-characters such as U + FFFF, there are also many UTF-16 code unit sequences that are not valid characters due to "surrogate approaches to encoding characters outside the 16-bit range. However, IsTextUnicode
is unaware of this since it predates astral planes.)
a source to share
Every character you come across is part of Unicode. For example, Latin 'a'
is U+0061
. This is especially true for Windows, which uses Unicode and UTF-16 encodings.
The Microsoft feature IsTextUnicode
is named rather unfortunately. It can be more accurately described as GuessTextEncodingFromRawBytes()
. I suspect your real problem is not with the interpretation of the raw bytes as you already know one character for it.
a source to share
I think you are mixing two different concepts. The character and its encoding do not match. Some characters (for example A
) are encoded the same way in ASCII or Latin-1 and UTF-8, some are not, some can only be encoded in UTF-8, etc.
IsTextUnicode()
tries to guess the encoding from the raw byte stream.
If, on the other hand, you already have a representation of a character and want to know if it can be originally expressed as ASCII or Latin-1 or some other encoding, then you can actually look at the character range ( [\u0000-\u007F]
for ASCII).
Finally, there are some invalid codes (for example \uFFFE
) that are possible representations of bytes that are not allowed as Unicode characters. But I don't think this is what you are looking for.
a source to share