Gcc, UTF-8 and limits.h
My OS is Debian, my default language is UTF-8 and my compiler is gcc. The default CHAR_BIT in limits.h is 8, which is normal for ASCII because ASCII has 1 char = 8 bits. But since I am using UTF-8, characters can be up to 32 bits, which contradicts the default CHAR_BIT of 8.
If I change CHAR_BIT to 32 in limits.h to better match UTF-8, what do I need to do for this new value to take effect? Think I need to recompile gcc? Do I need to recompile the Linux kernel? What about the default Debian packages installed, will they work?
a source to share
C and C ++ are defined char
as byte, i.e. an integer type for which it sizeof
returns 1. It shouldn't be 8 bits, but in the vast majority of cases it is. IMHO, it should have been called byte
. But in 1972, when C was created, Westerners didn't have to deal with multibyte character encodings, so you could get away with concatenating the "character" and "byte" types.
You just have to live with confusing terminology. Or from there. But don't edit the system header files. If you want to use character type instead of byte type use wchar_t
.
But UTF-8 string consists of 8 bit blocks of code, so it char
will work fine. You just need to remember the difference between char
and the character. For example, don't do this:
void make_upper_case(char* pstr)
{
while (*pstr != '\0')
{
*pstr = toupper(*pstr);
pstr++;
}
}
toupper('a')
works as expected, but toupper('\xC3')
- it's a pointless attempt at the uppercase half of a character.
a source to share
CHAR_BIT
- the number of bits in char
; never, never change it. It won't have the desired effect.
Work with char
s- encoded UTF-8 strings instead , or use strings wchar_t
if you want to store Unicode characters directly. *
* Small print: size wchar_t
also varies by system. On Windows with MSVC, this is only 16 bits, which is sufficient for the basic multilingual plane. You can use it with UTF-16 though, which works well with the Windows API. Most other systems wchar_t
give you 32 bits.
a source to share
You don't need char to be 32 bits for UTF-8 encoding. UTF-8 is a variable length encoding and is designed for 8-bit characters and is backward compatible with ascii.
You can also use wchar_t
, which is 32-bit (on Linux), but overall you wouldn't give much added value as handling Unicode is much more complex than just managing code points.
a source to share
I'm pretty sure CHAR_BIT is the number of bits in the type of the variable <char ', not the maximum number of bits in any character. As you can see, this is a constant in limits.h that does not change depending on the locale setting.
CHAR_BIT will equal 8 on any reasonably new / system system ... not 8 bit words these days :)
a source to share