Parsing large delimited files with dynamic number of columns
What would be the best approach to parsing a delimited file when the columns are unknown prior to parsing the file?
The file format is Rightmove v3 (.blm), the structure looks like this:
#HEADER#
Version : 3
EOF : '^'
EOR : '~'
#DEFINITION#
AGENT_REF^ADDRESS_1^POSTCODE1^MEDIA_IMAGE_00~ // can be any number of columns
#DATA#
agent1^the address^the postcode^an image~
agent2^the address^the postcode^^~ // the records have to have the same number of columns as specified in the definition, however they can be empty
etc
#END#
The files could potentially be very large, the example file I have is 40MB, but it could be several hundred megabytes. Below is the code I started before I realized the columns were dynamic, I open a stream when I read that this is the best way to handle large files. I'm not sure if my idea is to put each entry in the list, then the processing would be good, although I don't know if it would work with such large files.
List<string> recordList = new List<string>();
try
{
using (FileStream fs = new FileStream(path, FileMode.Open, FileAccess.Read))
{
StreamReader file = new StreamReader(fs);
string line;
while ((line = file.ReadLine()) != null)
{
string[] records = line.Split('~');
foreach (string item in records)
{
if (item != String.Empty)
{
recordList.Add(item);
}
}
}
}
}
catch (FileNotFoundException ex)
{
Console.WriteLine(ex.Message);
}
foreach (string r in recordList)
{
Property property = new Property();
string[] fields = r.Split('^');
// can't do this as I don't know which field is the post code
property.PostCode = fields[2];
// etc
propertyList.Add(property);
}
Any ideas on how to make this better? It's C # 3.0 and .Net 3.5, if that helps.
Thanks,
Anneli
a source to share
If you can cut some lines at the beginning (header content and # xxx # lines) then it is just a csv file with ^
as delimiter so any CSV reader class will do the trick.
a source to share
You can do this in several ways.
- If the properties of your objects have the same name as the columns in the data file, you can use reflection to determine which columns should be mapped to which properties.
- If the properties on your objects have different names, you can write a custom mapping scheme that says "for column X, assign property to Y".
- You can create custom attributes for object properties that indicate which column name they refer to, and use reflection to read those attributes.
All of these steps assume that the column names in your data files will be the same for the data they represent (ie, ADDRESS_1 will always be the column name for address bar one data).
a source to share