Data / data warehouse for large volume of simple transactions
I wrote a PHP application that requires millions of integers to be stored between 0 and 10,000,000, inclusive. Each number is increased by one very often (on average 100 values are updated every second) and read very often (20,000 views per second). The numbers are reset to 0 for either nightly, weekly, monthly, or yearly.
I have a pretty good MySQL tool, but it feels overkill and not very efficient in the process.
Has anyone had to deal with this before and / or could shed some light on a suitable storage system?
Depending on how much memory you have and how long you have to maintain them - you can just use APC cache or memcache. memcache has a nice boost, so it's insanely efficient at doing this.
By the way, you said that you need to read 20K. And do you do it in PHP? on a single server or on a cluster? Sounds like you need some kind of architectural guidance ..... :)
If this is a web application, you are going to use a cluster. And if it is not a web application, I would advise you to consider re-doing this in something that is not PHP. Java, C ++ and C # come to mind.
a source to share