Are hundreds / thousands of tcp sockets sane with memcached?
I am using Merb :: Cache to store txt / xml and noticed that the longer I leave my merbs, the more open tcp sockets I have, I believe this is causing some serious performance issues.
lsof | grep 11211 | wc -l 494
merb 27206 root 71u IPv4 13759908 TCP localhost.localdomain: 59756-> localhost.localdomain: 11211 (ESTABLISHED) merb 27206 root 72u IPv4 13759969 TCP localhost.localdomain: 59779-> localhost.localdomain: 11211 (ESTABLISHED) merb 27206 root 73u IPv4 13760039 TCP localhost.localdomain: 59805-> localhost.localdomain: 11211 (ESTABLISHED) merb 27206 root 74u IPv4 13760052 TCP localhost.localdomain: 59810-> localhost.localdomain: 11211 (ESTABLISHED) merb 27206 root 75u IPv4 13760135 TCP localhost.localdomain: 59841-> localhost.localdomain: 11211 (ESTABLISHED) merb 27206 root 76u IPv4 13760823 TCP localhost.localdomain: 59866-> localhost.localdomain: 11211 (ESTABLISHED) merb 27206 root 77u IPv4 13760951 TCP localhost.localdomain: 52095-> localhost.localdomain: 11211 (ESTABLISHED)
etc...
my relevant code:
if! exists? (: memcached) then
register (: memcached, Merb :: Cache :: MemcachedStore,: namespace => 'mynamespace',: servers => ['127.0.0.1:11211'])
end
& &
when: xml
unless @hand_xml = Merb :: Cache [: memcached] .read ("/ hands/# {@hand.id }.xml")
@hand_xml = display (@hand)
Merb :: Cache [: memcached] .write ("/ hands/# {@hand.id }.xml", @hand_xml)
end
return @hand_xml
Is this code wrong or am I using the wrong version of memcache ??
I have memcached 1.2.8 and have the following:
libmemcached-0.25.14.tar.gz Memcached-0.13.gem
it kind of drives me crazy ..
a source to share
k I figured out some things ..
1) it might be wise to have hundreds / thousands of sockets connected to memcached, assuming you are using a library that uses epoll or whatever - however, if you are using ruby like me, I am not aware of the lib that uses anything other than select () or poll (), so this immediately removes this question / immediately throws
2) if you are like me you only have 1 memcached server running right now and several mongrels / thins are running around taking care of requests ... so your memcache connections should be tried out. be no more than the number of mongrels / thins you have (assuming you only cache 1 or two sets of things), which was my case
here's the fix:
install memcache using memcached gem, not merb :: cache (which actually wraps whatever memcache library you use
MMCACHE = Memcached.new("localhost:11211")
get / set your values:
@cache = MMCACHE.clone
begin
@hand_xml = @cache.get("/hands/#{@hand.id}.xml")
rescue
@hand_xml = display(@hand)
@cache.set("/hands/#{@hand.id}.xml", @hand_xml)
end
@cache.quit
relax and drink a cold cause as you do:
lsof | grep 11211 | wc -l
you see something like 2 or 3 instead of 2036!
riff props for tying me to the fact that it's not uncommon for memcache connections to start with
a source to share
I could help, but I need to tell a story to do this. Here it is.
There used to be a cluster of 10 apache (ssl) servers configured to have exactly 100 threads each. There was also a cluster of 10 memcached servers (in the same boxes) and they all seemed to live peacefully. Both apache and memcached were guarded by the monit daemon .
Then King installed the 11th apache (ssl) server and memcached started restarting randomly every few hours! The king began an investigation and what did he find? There was an error in the php memcache module documentation that said the default constructor for the memcache connection object has no , but apparently it did. It so happened that every php thread (and there were about 1000 of them) opened a connection to every memcached in the pool when it needed it and it kept it. There were 10 * 100 connections to each memcached server and everything was fine, but with 11 servers it was 1100 and 1024 and 1100. The maximum number of open sockets for memcached was 1024. When all the sockets were made, the monit daemon could not connect, so it restarted memcached.
Every story must have a moral. So what did the king do with all this? He turned off the persistent connections, and they all lived happily ever after, with the number of connections per cluster peaking at 5 (five). These servers were serving a lot of data, so we couldn't have 1000 spare sockets and it was cheaper to negotiate a memcache connection on every request.
Sorry but I don't know ruby, it looks like you had an awful number of threads or you are using it incorrectly.
Good luck!
a source to share