I'm building a website with django that allows users to move content between a bunch of photography services. As you can imagine, the app makes a lot of api hits.

for example: a user connects picasa, flickr, photobucket and facebook to their account. Now we need to pull content from 4 different apis in order to update the data of these users.

right now I have a function that updates each api and I run them all at the same time via threads. (all api that are not included return false on the second line, there is not too much overhead to run them).

Here's my question:

What is the best strategy for keeping content up to date using these APIs?

I have two ideas that might work:

• Update the apis periodically (like a cron job) and all we have at the moment is what the user gets.

  Benefits

  :

  • Easy and simple to implement.
  • We will always have good data when the user loads their first page.

  traps:

  • we have to constantly do api attacks for users who are inactive, which wastes a lot of traffic.
  • The api providers are probably not happy.

• Run updates when user logs in (in a folder)

  advantages :

  • we save a bunch of bandwidth and run less risk of throwing api providers out.
  • NEVER requires the amount of resources on our servers.

  traps:

  • we either need to do the update asynchronously (and there will be nothing on the first login) or ...
  • The first page will take a very long time because we are fetching all the api data (so I measured 26 seconds).

edit : the design is very simple, the design only has two images, an external css file and two external javascript files.

Also, the number of 26 seconds comes from a Firebug network monitor running on a computer that was on the same local network as the server