Fast Crashes FastCgi - Want to catch all exceptions, but how?
I have a django app running on apache with fastcgi (uses Flup WSGIServer).
This gets set up via dispatch.fcgi, merged below:
#!/usr/bin/python
import sys, os
sys.path.insert(0, os.path.realpath('/usr/local/django_src/django'))
PROJECT_PATH=os.environ['PROJECT_PATH']
sys.path.insert(0, PROJECT_PATH)
os.chdir(PROJECT_PATH)
os.environ['DJANGO_SETTINGS_MODULE'] = "settings"
from django.core.servers.fastcgi import runfastcgi
runfastcgi(method="threaded",daemonize='false',)
Runfastcgi is the one that does the job and ultimately runs the WSGIServer on the WSGIHandler.
Occasionally, an exception occurs that causes fastcgi to crash.
EDIT: I don't know which error fastcgi crashes, or even fastcgi even crashes. I just know that sometimes the site goes down - consistently down - until the Apache reboots. The only errors that show up in the error.log are broken pipes and incomplete headers listed below.
Incomplete headers:
Note: I've replaced confidential information or confusion "..."
[Sat May 09 ...] [error] [client ...] (104)Connection reset by peer: FastCGI: comm with server ".../dispatch.fcgi" aborted: read failed
[Sat May 09 ...] [error] [client ...] FastCGI: incomplete headers (0 bytes) received from server ".../dispatch.fcgi"
[Sat May 09 ...] [error] [client ...] (32)Broken pipe: FastCGI: comm with server ".../dispatch.fcgi" aborted: write failed,
Broken pipe:
note: this happens for the trac site, not the django app, but it looks the same.
Unhandled exception in thread started by <bound method Connection.run of <trac.web._fcgi.Connection object at 0xb53d7c0c>>
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 654, in run
self.process_input()
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 690, in process_input
self._do_params(rec)
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 789, in _do_params
self._start_request(req)
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 773, in _start_request
req.run()
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 582, in run
self._flush()
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 589, in _flush
self.stdout.close()
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 348, in close
self._conn.writeRecord(rec)
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 705, in writeRecord
rec.write(self._sock)
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 542, in write
self._sendall(sock, header)
File "/usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 520, in _sendall
sent = sock.send(data)
socket.error: (32, 'Broken pipe')
I have looked in / var / log / apache 2 / error.log but I cannot find the reason for the failure. I sometimes have problems with swapping memory, but I think this is different. (Please excuse my ignorance. I am ready to learn how best to implement and debug the server admin.)
I would like to wrap runfastcgi with try / except. What's the best way to handle random exceptions (until I figure out the actual cause (s))?
I believe WSGIServer is handling a lot of requests. If I catch the exception, can I re-invoke runfastcgi without fear of an infinite loop? Should I return an HttpRequest error for the offending exception request? I don't even know how to do it.
I've been looking at django / core / servers / fastcgi.py and django / core / handlers / wsgi.py and django / http / init .py
I have not been able to make progress understanding what is happening.
Do I have ideas or experience?
Thanks!
This is probably a Flup bug . When a stream-based client connection is closed before flup finishes sending data, it raises a socket.error: (32, "Broken pipe") exception.
Trying to catch the exception while trying to catch around runfastcgi won't work. Just because the exception is being thrown by the thread.
OK, I'll explain why wrapping your own code in the catch attempt won't work. If you look closely at the exception trace, you can see that the first statement in the trace is not runfastcgi. This is because the exception is happening on a different thread. If you want to catch the exception, you need to wrap any of the statements specified in the trace in a try / catch like this:
# in file /usr/lib/python2.4/site-packages/Trac-0.12dev_r7715-py2.4.egg/trac/web/_fcgi.py", line 654, in run
try:
self.process_input()
except socket.error:
# ignore or print an error
pass
The point is that you can catch the error by changing the Flup code. But I don't see any benefit from this. Especially because this exception seems harmless and there is already a patch for it.
a source to share
A broken pipe is usually not determinable. You get a damaged pipe if the write operation on the pipe or socket fails because the other end has closed the connection. Therefore, if your FastCGI receives a Broken pipe, it means that the web server is closed for connection too early. In some cases, this is not a problem, it can be ignored silently.
As a quick hack, try catching and ignoring socket.error
with Broken pipe
. You may need to add a sentence except:
to many other places.
a source to share