Have Geek Brain

Python: sending a large dictionary to the server

I have an application that needs to send state information to a server. This information is actually a large dictionary of string keys.

The server will run the Turbogears based web app, so the server side method takes an arbitrary number of keyword arguments.

In addition to the actual data, some authentication related data (id, password ..) needs to be passed. One approach would be to simply cast a pointer to a large dictionary containing it all and send it in a request to the server.

urllib2.urlencode(dataPlusId)

But in reality, the method that authenticates and accepts the dataset doesn't need to know much about the data. Data can be transmitted and received transparently and transferred to another data-handling method.

So my question is, what is the best way to transfer a large data dictionary to the server as a whole? And in this particular case, what's the best way to handle authentication here?

0


a source to share


6 answers


I agree with all the answers about avoiding the pickle if security is an issue (maybe not if the sender is authenticated before the data is scattered around), but with security, two layers of protection might be better than one); JSON often helps in such cases (or XML if it doesn't do anything ...! -).



Authentication should ideally be left on the web server as SpliFF recommends, and SSL (i.e. HTTPS) is generally good for this. If this is not feasible, but it is entirely possible that the client and server will share "secrets", then sending the serialized string encrypted might be best.

+4


a source


I think the best way is to encode your data in an appropriate transfer format (you shouldn't use pickle as it doesn't store, but can be binary) and pass it as a multipart post request

What I don't know is if you can get it to work with repoze.who. If it doesn't support one-step login and function call, you might have to verify the credentials yourself.



If you can wrap your data in xml, you can use XML-RPC as well .

+3


a source


Why don't you serialize the dictionary to a file and load the file? This way the server can read the object back into the dictionary.

+2


a source


Do a POST of your python data (use binaries as suggested in other answers) and handle security with your webserver. Apache and Microsoft servers can authenticate using a wide variety of methods (SSL client certificates, password, system accounts, etc.)

Serializing / deserializing text or XML is probably too big if you go back to the dictionary again).

+2


a source


I personally used SimpleJSON at both ends and would just post the "file" (it would just be a stream) as multipart data.

But that's me. There are other options as well.

+2


a source


Have you tried using pickle for data?

+1


a source






More articles:


All Articles