Most elegant way to split CSV columns into separate data structures using Python?
I am trying to pick up Python. As part of the learning process, I am migrating a project I wrote in Java to Python. I'm currently in the section where I have a list of CSV form headers:
headers = [a, b, c, d, e, .....]
and separate lists of groups into which these headers should be split, for example:
headers_for_list_a = [b, c, e, ...] headers_for_list_b = [a, d, k, ...] . . .
I want to take CSV data and turn it into a dict based on these groups, for example:
list_a = [ {b:val_1b, c:val_1c, e:val_1e, ... }, {b:val_2b, c:val_2c, e:val_2e, ... }, {b:val_3b, c:val_3c, e:val_3e, ... }, . . . ]
where, for example, val_1b is the first row of column "b", val_3c is the third row of column "c", and so on.
My first "Java instinct" is to do something like:
for row in data:
for col_num, val in enumerate(row):
col_name = headers[col_num]
if col_name in group_a:
dict_a[col_name] = val
elif headers[col_cum] in group_b:
dict_b[col_name] = val
...
list_a.append(dict_a)
list_b.append(dict_b)
...
However, this method seems inefficient / cumbersome and lacks the finesse that Python programmers talk about all the time. Is there a more "Zen-like" way that I should stick to the Python philosophy?
a source to share
csv.DictReader
import csv
groups = dict(a=headers_for_list_a, b=headers_for_list_b)
lists = dict((name, []) for name in groups)
for row in csv.DictReader(csvfile, fieldnames=headers):
for name, grp_headers in groups.items():
lists[name].append(dict((header, row[header]) for header in grp_headers))
a source to share
You don't need the most pythonic way to achieve the same as your code, but this version of your code is somewhat more concise due to the use of generator expressions:
from itertools import izip
for row in data:
dict_a = dict((col_name, val) for col_name, val in izip(headers, row) \
if col_name in group_a)
dict_b = dict((col_name, val) for col_name, val in izip(headers, row) \
if col_name in group_b)
list_a.append(dict_a)
list_b.append(dict_b)
Also, instead of sets, use sets group_a
and group_b
- the operator in
is faster on sets. But Jason Humber is right, DictReader
is more elegant, see the next version:
from csv import DictReader
for row in DictReader(your_file, headers):
dict_a = dict((k, row[k]) for k in group_a)
dict_b = dict((k, row[k]) for k in group_b)
list_a.append(dict_a)
list_b.append(dict_b)
a source to share