Squid magazines on mongodb
I plan on recording my squid instances to mongodb, but the actual problem is that we have huge traffic to register, every access is authenticated with user / pass. At the end of the day, we have to do some log-based reports. I was thinking about inserting magazines distributed by months and users, so my collection would look like this:
{month: 'april', users: [{user: 'loop0', logs: [{timestamp: 12345678.9, url: 'http://stackoverflow.com/question/ask', ... }]}]
So if I want to generate my reports based on the month of April, I just need to get the correct month instead of searching through the cillins of the rows to get the rows corresponding to the timestamp between April 1st and April 30th.
Of course, this type of insert will be slower than just inserting the log line directly. So my question is, is there a better way to do this?
We currently have about 12 million log lines per day.
a source to share
It's hard to tell without knowing the details, but I would say that you are probably worried about the wrong problem: you are thinking about insertion speed, not report computation speed.
Mongo has all day to store these 12 million records, but you might want a report covering maybe half a billion records (~ = 1 month) - real-time rendering (seconds, maybe a minute). From this point of view, it is probably desirable to optimize reading, not writing.
a source to share