Problem using getline with unicode file
UPDATE: Thanks to @Potatoswatter and @Jonathan Leffler for the comments - rather embarrassingly I was caught by the debugger tooltip not showing the wstring value correctly - however it still doesn't quite work for me and I have an updated question below:
If I have a small multibyte file that I want to read in a line, I use the following trick: I use getline
with a divisor '\0'
, eg.
std::string contents_utf8;
std::ifstream inf1("utf8.txt");
getline(inf1, contents_utf8, '\0');
This is read across the entire file, including newlines.
However, if I try to do the same with a wide format file, it doesn't work - mine wstring
only reads the first line.
std::wstring contents_wide;
std::wifstream inf2(L"ucs2-be.txt");
getline( inf2, contents_wide, wchar_t(0) ); //doesn't work
For example, my if unicode file contains characters A and B separated by CRLF, hexadecimal looks like this:
FE FF 00 41 00 0D 00 0A 00 42
Based on the fact that with a multibyte file, getline with '\ 0' reads the entire file, I figured I getline( inf2, contents_wide, wchar_t(0) )
should read the entire Unicode file. However, this is not the case - with the example above, my wide string will contain the following two wchar_ts:FF FF
(If I remove wchar_t (0) it reads on the first line as expected (i.e. FE FF 00 41 00 0D 00
)
Why does wchar_t (0) not work as a wchar_t delimiter so getline stops at 00 00
(or reads at the end of the file, which is what I want)?
thank you
a source to share
Your UCS-2 decoder is wrong. Result getline( inf2, contents_wide )
on FE FF 00 41 00 0D 00 0A 00 42
must be 0041 0000
= L"A"
. Assuming you are on Windows, line endings should be correctly converted and no byte sign should appear in the output.
Suggest that you double check your operating system's documentation for how you set the locale.
EDIT: Have you set the locale?
locale::global( locale( "something if your system supports UCS-2" ) );
or
locale::global( encoding_support::ucs2_bigendian_encoding );
where encoding_support is some library.
a source to share
Have a look at this question: Why is the widespread file stream in C ++ narrow written data by default? Where poster surprised that wchar_t
→ char
conversion during recording.
The answers to this question also apply to the reading case. In a nutshell: at the lowest level, file I / O is always done in bytes. A basic_filebuf
(which is what it uses fstream
to actually do the I / O) uses a facet codecvt
to translate between "internal" encoding (a char type visible to the program and used to instantiate the stream, wchar_t
in your case) and the "external" encoding of the file (which is always char
) ...
codecvt
is obtained from the stream locale
. If imbue()
-d is not specified on the stream , the global locale is used. By default, the global locale is the "classic" (or "C") language. This language style is codecvt
pretty simple. I don't know what the standard says about this, but in my experience on Windows, it just "casts" between char
and wchar_t
one after the other. On Linux it does this too, but does not work if the character value is outside the ASCII range.
So, if you are not concerned with the locale (either imbue()
-into one in the stream or changing the global), then what is probably happening in your case is that it char
reads from the file and executes up to wchar_t
one by one . So it first reads FF
, then then FE
, 00
and getline(..., 0)
stops right there.
a source to share