Document Actions
Using The Deadlock Debugger
If you ever find your Zope using 99% of CPU or otherwise not performing as well as you would like, then DeadlockDebugger is for you. Nuxeo's DeadlockDebugger will show you at any time what your Zope's threads are spending precious time on, and thereby enable you to quickly track down and resolve problems. At HSR, we regularly use DeadlockDebugger to decipher complex Zope software problems - and so should you.
DeadlockDebugger is a Normal Zope Product
DeadlockDebugger is mostly like any other Zope Product, so the first step is to login via SSH and go to the Products directory of your Zope instance.
Installation
You should become root and then:
su zope cd ~/instance1/Products
Now download and unpack DeadlockDebugger:
wget http://www.zope.org/Members/nuxeo/Products/DeadlockDebugger/1.0/DeadlockDebugger-1.0.tgz tar xzf DeadlockDebugger-1.0.tgz rm DeadlockDebugger-1.0.tgz
Configuration
You'll want to go into the resulting DeadlockDebugger directory and edit a file, but first, get a secure password suggestion:
genpass
Now:
cd DeadlockDebugger vi custom.py
Now set the "ACTIVATED" variable to "True" and change the value for "SECRET" to the secure password you selected:
ACTIVATED = True SECRET = 'SECURE_PASSWORD' # you must change this
Save the file and you are done installing DeadlockDebugger.
The threadframe Python Module
In order for DeadlockDebugger to do its job, you need to install the threadframe python module.
First, download it:
cd wget http://www.majid.info/python/threadframe/threadframe-0.2.tar.gz tar xzf threadframe-0.2.tar.gz rm threadframe-0.2.tar.gz cd threadframe-0.2
You can now install it. First, you need to become root:
exit cd /usr/local/zope/threadframe-0.2 python2.3 setup.py install
Restart Zope
Now you can restart your Zope:
zopectl restart
and navigate to this URL:
http://url_to_my_zope_root/manage_debug_threads?SECURE_PASSWORD
Where "url_to_my_zope_root" could be your temporary domain or hostname and SECURE_PASSWORD is the secure password you entered in the custom.py file.
Watch your Threads
Assuming all is well, and your Zope is using the default four threads, you may see something like this:
Threads traceback dump at 2005-06-08 10:54:34
Thread 185854976:
File "/usr/local/zope2/2701/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__
name, request, response=accept()
File "/usr/local/zope2/2701/lib/python/ZServer/PubCore/ZRendezvous.py", line 45, in accept
l.acquire()
Thread 185853952:
File "/usr/local/zope2/2701/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__
name, request, response=accept()
File "/usr/local/zope2/2701/lib/python/ZServer/PubCore/ZRendezvous.py", line 45, in accept
l.acquire()
Thread 185852928:
File "/usr/local/zope2/2701/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__
name, request, response=accept()
File "/usr/local/zope2/2701/lib/python/ZServer/PubCore/ZRendezvous.py", line 45, in accept
l.acquire()
Thread 184738816:
File "/usr/local/zope2/2701/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__
name, request, response=accept()
File "/usr/local/zope2/2701/lib/python/ZServer/PubCore/ZRendezvous.py", line 45, in accept
l.acquire()
End of dump
The page does not automatically refresh, so you will have to do this with your browser's refresh function at the desired frequency.
If you see something else for one or more of your threads it may be normal, but if you are constantly seeing the same thing while your Zope's CPU usage skyrockets, then you are probably looking at a very large clue to the source of your Zope's problem(s).
When You are Done ...
When you are done using DeadlockDebugger for the current session, you should edit custom.py again, turn ACTIVATED to False and then restart your Zope. This will ensure that an attacker cannot exploit the debugging information to do you harm.
