Accidentally long DB / Memcache fetch queries into production env

I am having trouble diagnosing a problem I am having in my ubuntu scalr / ec2 production environment.

The problem seems to be random, database queries and / or memcache queries will consume a LOT longer than necessary. I've seen that a simple select statement takes 130ms, or a selection from Memcache takes 65ms! This can happen multiple times per request, resulting in some requests taking twice as long as necessary.

To diagnose the problem, I wrote a very simple script that would simply connect to the MySql server and run a query.

require 'mysql'

mysql = Mysql.init
mysql.real_connect('', '', '', '')

max = 0
100.times do
  start = Time.now
  mysql.query('select * from navigables limit 1')
  stop = Time.now

  total = stop - start
  max = total  if total > max
end

puts "Max Time: #{max * 1000}"

mysql.close

      

        
        
        
      

    

This script was consistently returning really high maximum times, so I eliminated any Rails as the source of the problem. I also wrote the same thing in Python to exclude Ruby. Indeed, Python was taking too long too!

Both MySql and Memcache are in their own boxes, so I was looking at network latency, but browsing ping

and traceroute

ingesting looked ok.

Also doing queries / fetching on the respective machines returns the expected time and I am running the same versions on my staging machine without this issue.

I'm really fixated on this ... any thoughts of something I could try to diagnose? Thanks to