What is the Least Resource Structure for Distribution with a Python Application
I am creating an application for distribution to other scientists. The application will accept three parameters that the user presents and displays a list of dates and codes associated with these events. I was building this with a dictionary and was going to create an application so that the dictionary is loaded from the pickle file when the application called it. The parameters provided by the user will be used to find the desired exit.
I chose this framework because I got pretty comfortable with dictionaries and pickled files and I can see it coming out the door with the smallest learning curve on my end. A dictionary can have up to two million keys. I was happy with the performance on my machine with a reasonable subset. I've already figured out how to split the dictionary in case I have performance issues when the whole thing is compiled. I'm not really worried about the amount of disk space on my machine since we're working with terabyte storage values.
Having said all this, I've already said it in the docs and I'm wondering if I need to spend some time researching and implementing an alternative data storage file. The only reason I can think of is if there is an alternative that can increase the search speed by three to five times or more.
a source to share
The standard module shelve
will provide you with a persistent dictionary that is stored in the dbm style database. If your keys are strings and your values are picklable (since you are using pickle already, that should be true), this might be the best solution that just keeps the entire dictionary in one pickle.
Example:
>>> import shelve
>>> d = shelve.open('mydb')
>>> d['key1'] = 12345
>>> d['key2'] = value2
>>> print d['key1']
12345
>>> d.close()
I would also recommend Durus , but it requires additional training on your part. This will allow you to create a PersistentDictionary. From memory, keys can be any mocked object.
a source to share
For a quick search, use the standard Python module dbm
(see http://docs.python.org/library/dbm.html ) to create a database file and search within it. The dbm file format may not be cross-platform, so you might want to distribute your data in Pickle or Rep, JSON or YAML, or XML format, and create a database dbm
where the user runs your program.
a source to share
How much memory can your application use? Will this run on every user's desktop or will there only be one deployment somewhere?
An in-memory python dictionary can probably handle two million keys for sure. You say that you have a subset of the data; do you have a lot? Maybe you should throw the full dataset into it and see if it handles.
I just tested the creation of a 2 million dictionary; the total memory usage for the process was about 200MB. If speed is your main concern and you have RAM, then you probably won't do better than an in-memory python dictionary.
a source to share
Check out this solution on SourceForge, especially. endnotes documentation:
y_serial.py module :: Python Object Store with SQLite
"Serialization + persistence :: in a few lines of code, compress and annotate Python objects in SQLite, and then chronologically restore them by keywords without any SQL. Most useful" standard "database module for storing schema-less data."