Parsing a text file in C # while skipping some content
I am trying to parse a text file with header and body. In the header of this file, there are links to line numbers for body sections. For instance:
SECTION_A 256
SECTION_B 344
SECTION_C 556
This means SECTION_A starts on line 256.
What would be the best way to parse this heading in a dictionary and then read the sections as needed.
Typical scenarios:
- Parse the header and read-only section SECTION_B
- Parse the heading and read the first paragraph of each section.
The data file is quite large and I definitely don't want to load it all into memory and then work with it.
I would be grateful for your suggestions. My environment is VS 2008 and C # 3.5 SP1.
a source to share
Well, obviously you can store the name + line number in the dictionary, but that won't do you any good.
Well, of course, this will let you know which line to start reading, but the problem is where in the file that line is located? The only way to know is to start over and start counting.
The best way would be to write a wrapper that decodes the text content (if you have encoding problems) and can provide you with the line number for the bytes of the collation type position, then you can take that line number, 256 and look in the dictionary to know that line 256 starts at position 10000 in the file and starts reading from there.
Is this a one-time processing situation? If not, do you think the entire file is loaded into a local database like SQLite database? This will allow you to have a direct mapping between the line number and its content. Of course this file will be even larger than your original file and you will need to copy the data from the text file into the database, so there is some overhead anyway.
a source to share
You can do this quite easily.
There are three parts to this problem.
1) How to find where a line starts in a file. The only way to do this is to read lines from a file, storing a list that records the starting position in the file of that line. eg
List lineMap = new List ();
lineMap.Add (0); // Line 0 starts at location 0 in the data file (just a dummy entry)
lineMap.Add (0); // Line 1 starts at location 0 in the data file
using (StreamReader sr = new StreamReader ("DataFile.txt"))
{
String line;
int lineNumber = 1;
while ((line = sr.ReadLine ())! = null)
lineMap.Add (sr.BaseStream.Position);
}
2) Read and parse your index file in the dictionary.
Dictionary index = new Dictionary ();
using (StreamReader sr = new StreamReader ("IndexFile.txt"))
{
String line;
while ((line = sr.ReadLine ())! = null)
{
string [] parts = line.Split (''); // Break the line into the name & line number
index.Add (parts [0], Convert.ToInt32 (parts [1]));
}
}
Then, to find a line in your file, use:
int lineNumber = index ["SECTION_B";]; // Convert section name into the line number long offsetInDataFile = lineMap [lineNumber]; // Convert line number into file offset
Then open a new FileStream at DataFile.txt, Seek (offsetInDataFile, SeekOrigin.Begin) to go to the beginning of the line, and use a StreamReader (as above) to read lines from it.
a source to share
Read the file to the end of the header, assuming you know where it is. Separate the lines you saved in space, like:
Dictionary<string, int> sectionIndex = new Dictionary<string, int>();
List<string> headers = new List<string>(); // fill these with readline
foreach(string header in headers) {
var s = header.Split(new[]{' '});
sectionIndex.Add(s[0], Int32.Parse(s[1]));
}
Find the dictionary entry you want, store the number of lines read in the file, and loop until you hit that line number, then read until you get to the next starting line of the section. I don't know if you can guarantee the order of the keys in the dictionary, so you probably need the names of the current and next sections.
Be sure to do some error checking to make sure the section you are reading is not before the section you are reading and any other errors you may think of.
a source to share
You can read line by line until all of the header information is captured and stopped (assuming all section pointers are in the header). You will have section and line numbers to use when retrieving data later.
string dataRow = "";
try
{
TextReader tr = new StreamReader("filename.txt");
while (true)
{
dataRow = tr.ReadLine();
if (dataRow.Substring(1, 8) != "SECTION_")
break;
else
//Parse line for section code and line number and log values
continue;
}
tr.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
a source to share