Updating local sqlite database that is used for local metadata and caching from service?

I browsed the site and did not find a question / answer that quite answered my question, the closest one I found is: Synchronizing objects between two disparate systems is the best approach .

Anyway, to get started, because there are no rss feeds available, I browse the web page so it makes a fetch, then browses the web page to discard all the information I'm interested in and dump its information into a sqlite database so I can request information at my leisure without having to reselect from the website.

However, I also store various metadata in the data itself, which is stored in the sqlite db, for example: I looked at the data, this data is new / old, bookmarks for a chunk of data (think of it as a set of unrelated data, and a bookmark is just a pointer to something, where I am in processing / reading said data).

So right now my current problem is trying to figure out how to update the local sqlite database with new data and / or changed data from the website in a way that is efficient and simple.

Here is my current idea:

  • Loading the page itself
  • Create a temp table for the parsed data to go to
  • Match the official and temporary table and copy updates and / or new information to the official table.

This process seems quite complicated because I need to figure out how to determine if the data in the temporary table is new, updated, or immutable. So I am wondering if there is no better approach or does anyone have any suggestions on how to architecture / structure such a system?

Edit 1: I'm not sure where to put the additional information, in a comment or as an edit, so I'm going to add it here.

This expands the metadata a bit with regard to bookmarks, basically the data source can create new data / addition to the current data, so one of the reasons why I was thinking about creating a temporary table was so that I could to determine if the data source has which was "bookmarked", any new data or not.

+2


a source to share


1 answer


Is it really important to determine if the data in the temporary table is new, updated, or unchanged? Do you really need to keep a change history?

NO: don't use a temp table, just put (timestamp) your old records, don't make updates and just insert your new data.



YES: your idea seems correct to me, but it all depends on how much data you need to process each time; I don't think this is possible with a lot of data.

0


a source







All Articles