Debugging a Memory LeakΒΆ
To see the kind of power Eww gives you, we’re going to use it to diagnose a memory leak.
This script here leaks memory quite badly:
#! /usr/bin/env python
class Parent(object):
def __init__(self):
self.child = None
def __del__(self):
pass
class Child(object):
def __init__(self):
self.parent = None
def __del__(self):
pass
if __name__ == '__main__':
for num in range(500):
parent = Parent()
child = Child()
parent.child = child
child.parent = parent
If only all reference cycles were this simple.
Note
Why does this leak?
Python uses a reference counting scheme for reaping old objects. Python keeps track of the number of names attached to objects, and if that number drops to 0, Python will reap that object.
Python can also handle most reference cycles. The gc (garbage collection) module will take care of that for us.
What the gc can’t do is fix reference cycles where both objects have a __del__ method. The gc cannot automatically determine a safe order to run them in, so it refuses to reap either object.
This assumes you are using the CPython implementation. The Python specification does not mention anything about object management, and individual implementations (PyPy, Jython, IronPython) may use different garbage collection techniques.
Let’s, for the exercise, say you’ve been running this in production for a while and notice that memory usage is constantly growing.
We’ll add Eww to the script and see what we can find out.
Add import eww to the script:
#! /usr/bin/env python
import eww
Then we’ll make two small changes to the code creating the objects:
if __name__ == '__main__':
eww.embed() # 1
for num in range(500):
parent = Parent()
child = Child()
parent.child = child
child.parent = parent
import time; time.sleep(600) # 2
We added a call to eww.embed(), which sets up everything for Eww. We also added a sleep at the end so the script doesn’t immediately exit and we can look at what’s going on. Go ahead and run the script.
Now, we’ll fire up the Eww client. In another terminal window, run eww. You should see something like this:
basecamp ~: eww
Welcome to the Eww console. Type 'help' at any point for a list of available commands.
Running in PID: 4899 Name: ./leak_demo.py
(eww)
The Eww client connects to the Eww console running inside your application. We can now do just about anything we want, while we’re inside the running app.
Let’s get a REPL:
(eww) repl
Dropping to REPL...
Python 2.7.5 (default, Mar 9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Note: This interpreter is running *inside* of your application. Be careful.
>>>
Since we’re in the same process as the leaky script, we can check for uncollectable cycles very easily:
>>> import gc
>>> gc.collect()
0
>>> len(gc.garbage)
998
>>>
998 uncollectable objects. Ouch.
Before we dig any deeper, we ought to get some statistics around memory consumption so we can verify the bug and our fix.
To do that, we’ll use Eww’s statistics and graphing tools. Let’s add a datapoint at the start of each iteration in the for loop:
for num in range(500):
eww.graph('Memory Usage', (num, eww.memory_consumption()))
parent = Parent()
Restart the leaky script, and connect with the Eww client again. This time, instead of going straight to the REPL, let’s check out our new stat:
(eww) stats
Graphs:
Memory Usage:500
(eww)
Cool, we’ve got 500 datapoints for the ‘Memory Usage’ statistic. We can get the raw datapoints by running stats 'Memory Usage', but that’s not very helpful. Let’s generate a graph instead:
(eww) stats -g 'Memory Usage'
Chart written to Memory Usage.svg
(eww)
Which gives us something like this:
Yep, that’s a memory leak.
To identify exactly what’s causing the leak, we’re going to use Objgraph. A simple pip install objgraph will install it for you. You’ll also need to have graphviz installed for the nice graphs.
Note
Objgraph includes a method called show_most_common_types() that can be used to find leaking objects. We are inspecting the gc.garbage list instead, but it’s good to be familiar with show_most_common_types().
Time to dig into one of the uncollectable objects in gc.garbage and see if we can find the issue. Let’s head into the REPL again and use objgraph to show us some more detail:
(eww) repl
Dropping to REPL...
Python 2.7.5 (default, Mar 9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Note: This interpreter is running *inside* of your application. Be careful.
>>> import gc
>>> gc.collect()
0
>>> import objgraph
>>> objgraph.show_backrefs(gc.garbage[0], max_depth=10)
Graph written to /var/folders/gv/1xcz06bj2c5gfkl632fbjfr40000gn/T/objgraph-_qnjjD.dot (9 nodes)
Graph viewer (xdot) not found, generating a png instead
Image generated as /var/folders/gv/1xcz06bj2c5gfkl632fbjfr40000gn/T/objgraph-_qnjjD.png
>>>
The generated graph makes the problem crystal clear:
There are a few ways to fix this.
- We can use the weakref module to make Child’s reference to Parent a weak reference.
- We can explicitly break the reference.
- We can change the scripts design to not require cyclic references like this.
Let’s go with Option #2 for simplicity here. Update the for loop to have a del at the end of each iteration:
for num in range(500):
eww.graph('Memory Usage', (num, eww.memory_consumption()))
parent = Parent()
child = Child()
parent.child = child
child.parent = parent
del child.parent
It’s easy to tell if this worked. We’ll connect with Eww, check if there are any uncollectable objects, and graph memory usage again:
(eww) repl
Dropping to REPL...
Python 2.7.5 (default, Mar 9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Note: This interpreter is running *inside* of your application. Be careful.
>>> import gc
>>> gc.collect()
0
>>> len(gc.garbage)
0
>>> # Good!
>>> exit
Exiting REPL...
(eww) stats -g 'Memory Usage' -f 'memory_usage_after_fix'
Chart written to memory_usage_after_fix.svg
(eww)
When we look at the new graph, we don’t see a perfectly flat line like we’d expect. This has to do with how Pymalloc works (and some memory consumption by Eww). What’s important is to compare the scale of this graph to the scale of the previous graph.
That’s about it for the basic tour. Read on for more advanced features, and information on how Eww works.