Updating local sqlite database that is used for local metadata and caching from service?
I browsed the site and did not find a question / answer that quite answered my question, the closest one I found is: Synchronizing objects between two disparate systems is the best approach .
Anyway, to get started, because there are no rss feeds available, I browse the web page so it makes a fetch, then browses the web page to discard all the information I'm interested in and dump its information into a sqlite database so I can request information at my leisure without having to reselect from the website.
However, I also store various metadata in the data itself, which is stored in the sqlite db, for example: I looked at the data, this data is new / old, bookmarks for a chunk of data (think of it as a set of unrelated data, and a bookmark is just a pointer to something, where I am in processing / reading said data).
So right now my current problem is trying to figure out how to update the local sqlite database with new data and / or changed data from the website in a way that is efficient and simple.
Here is my current idea:
- Loading the page itself
- Create a temp table for the parsed data to go to
- Match the official and temporary table and copy updates and / or new information to the official table.
This process seems quite complicated because I need to figure out how to determine if the data in the temporary table is new, updated, or immutable. So I am wondering if there is no better approach or does anyone have any suggestions on how to architecture / structure such a system?
Edit 1: I'm not sure where to put the additional information, in a comment or as an edit, so I'm going to add it here.
This expands the metadata a bit with regard to bookmarks, basically the data source can create new data / addition to the current data, so one of the reasons why I was thinking about creating a temporary table was so that I could to determine if the data source has which was "bookmarked", any new data or not.
a source to share
Is it really important to determine if the data in the temporary table is new, updated, or unchanged? Do you really need to keep a change history?
NO: don't use a temp table, just put (timestamp) your old records, don't make updates and just insert your new data.
YES: your idea seems correct to me, but it all depends on how much data you need to process each time; I don't think this is possible with a lot of data.
a source to share