Optimise Python with closures
This blog post by Dan Crosta is interesting. It talks about how is possible to optimise Python code for operations that get called multiple times avoiding the usage of Object Orientation and using Closures instead.
While the “closures” gets the highlight, the main idea is a little more general. Avoid repeating code that is not necessary for the operation.
The difference between the first proposed code, in OOP way
class PageCategoryFilter(object):
def __init__(self, config):
self.mode = config["mode"]
self.categories = config["categories"]
def filter(self, bid_request):
if self.mode == "whitelist":
return bool(
bid_request["categories"] & self.categories
)
else:
return bool(
self.categories and not
bid_request["categories"] & self.categories
)
and the last one
def make_page_category_filter(config):
categories = config["categories"]
mode = config["mode"]
def page_category_filter(bid_request):
if mode == "whitelist":
return bool(bid_request["categories"] & categories)
else:
return bool(
categories and not
bid_request["categories"] & categories
)
return page_category_filter
The main differences are that both the config dictionary and the methods (which are also implemented as a dictionary) are not accessed. We create a direct reference to the value (categories and mode) instead of making the Python interpreter search on the self methods over and over.
This generates a significant increase in performance, as described on the post (around 20%).
But why stop there? There is another clear win in terms of access, assuming that the filter doesn’t change. This is the “mode”, which we are comparing for whitelist of blacklist on each iteration. We can create a different closure depending on the mode value.
def make_page_category_filter2(config):
categories = config["categories"]
if config['mode'] == "whitelist":
def whitelist_filter(bid_request):
return bool(bid_request["categories"] & categories)
return whitelist_filter
else:
def blacklist_filter(bid_request):
return bool(
categories and not
bid_request["categories"] & categories
)
return blacklist_filter
There are another couple of details. The first one is to transform the config categories into a frozenset. Assuming that the config doesn’t change, a frozenset is more efficient than a regular mutable set. This is insinuated in the post, but maybe didn’t get the final review (or to simplify it).
Also, we are calculating the intersection of a set (operand &) to then reduce it to a bool. There is currently a set operation that gets the result without calculating the whole intersection (isdisjoint).
The same basic principle applies to calculate the bool category for the black filter. We can calculate it only once, as it’s there to short-circuit the result in case of an empty config category.
def make_page_category_filter2(config):
categories = frozenset(config["categories"])
bool_cat = bool(categories)
if config['mode'] == "whitelist":
def whitelist_filter(bid_request):
return not categories.isdisjoint(bid_request["categories"])
return whitelist_filter
else:
def blacklist_filter(bid_request):
return (bool_cat and categories.isdisjoint(bid_request["categories"]))
return blacklist_filter
Even if all of this enters the definition of micro-optimisations (which should be used with care, and only after a hot spot has been found), it actually makes a significant difference, reducing the time around 35% from the closure implementation and ~50% from the initial reference implementation.
All these elements are totally applicable to the OOP implementation, by the way. Python is quite flexible about assigning methods. No closures!
class PageCategoryFilter2(object):
''' Keep the interface of the object '''
def __init__(self, config):
self.mode = config["mode"]
self.categories = frozenset(config["categories"])
self.bool_cat = bool(self.categories)
if self.mode == "whitelist":
self.filter = self.filter_whitelist
else:
self.filter = self.filter_blacklist
def filter_whitelist(self, bid_request):
return not bid_request["categories"].isdisjoint(self.categories)
def filter_blacklist(self, bid_request):
return (self.bool_cat and
bid_request["categories"].isdisjoint(self.categories))
Show me the time!
Here is the updated code, adding this implementations to the test.
The results in my desktop (2011 iMac 2.7GHz i5) are
total time (sec) time per iteration
class 9.59787607193 6.39858404795e-07
func 8.38110518456 5.58740345637e-07
closure 7.96493911743 5.30995941162e-07
class2 6.00997519493 4.00665012995e-07
closur2 5.09431600571 3.39621067047e-07
The new class performs better than the initial closure! The optimised closure is anyway trumping, saving a big chunk compared with the slower implementation. The PyPy results are all very close, and it speeds up 10x the code, which is an amazing feat.
Of course, a word of caution. The configuration is assumed to not change for a filter, which I think is reasonable.
Happy optimising!
Wrong Side of Memphis 
Hi Jamie, thanks for this. The code samples in my post are, as you suspected, somewhat simplified from the actual code we run in production, for the sake of demonstrating the point in a blog. In particular, we’d already come to the same conclusion as you have about returning one of two functions as the closure.
We also have sentinel values `always_allow` and `always_block` for configurations with degenerate cases, which simplifies out the `bool_cat` that you are showing here. The code that generates the filters has special handling of these values which either prevent the filter from ever being called, or remove the campaign from the list of considered campaigns entirely.
I hadn’t thought to use the other methods on sets rather than intersection, that’s a great tip. Looks like it saves us about 30% in CPython, and about 3% in PyPy. Nice!
Isn’t the takeaway “just use pypy” ?
It depends. In this case (mostly number crunching), for sure.
If your code is waiting for I/O (which is a pretty typical case), then PyPy is not going to solve much.
And PyPy has also other problems, like need to warmup (bad for processes that run once and then exit) and possible incompatibilities if there are C extensions involved.
But PyPy is here to stay, that’s for sure.
Pingback: dcrosta comments on “Optimize Python with closures” | Exploding Ads
One further optimization, again assuming the config doesn’t change: instead of saving frozenset(config[“categories”]), save frozenset(config[“categories”]).isdisjoint. That will give you a bound method that you can call directly, saving another lookup.
Clever. That also lets you name that bound method to something possibly more meaningful.