Have Geek Brain

Does Python module use memory mapped shelve IO?

Does anyone know if a Python shelve

IO module uses memory mapped memory?

Perhaps this question is a little misleading. I understand I am shelve

using a basic dbm style module to do my dirty work. What are the chances that the basic module is using mmap

?

I am prototyping a data warehouse, and while I understand that premature optimization is generally frowned upon, it can really help me understand the trade-offs associated with my design.

+2


source to share


2 answers


Existing implementations dbm

in the Python standard library use "normal" I / O, not memory mapping. You will need to link your own dbm

memory mapped ish implementation and integrate it with shelve

(directly or more productively via anydbm

).



+4


source


I'm not sure what you are trying to find out by asking this question, since you already know the answer: it depends on the dbm store you are using. Some of them will use mmap - I expect everyone but dumbdbm will use mmap - but what? The shelf overhead is almost certainly not in the mmap-versus-fileIO variant, but in the etching operation. You can't die in a dbm file yourself anyway, as the dbm module might have its own fancy locking (and it might not be a single file anyway, like when it uses bsddb.)



If you're just looking for inspiration for your own datastore, well, don't look at the shelf as all it does is pickle and go to another datastore.

+3


source






More articles:


All Articles