Updated on 2020-06-25: Replaced time.time() with time.perf_counter()

When I first learned about Python decorators, using them felt like doing voodoo magic. Decorators can give you the ability to add new functionalities to any callable without actually touching or changing the code inside it. This can typically yield better encapsulation and help you write cleaner and more understandable code. However, decorator is considered as a fairly advanced topic in Python since understanding and writing it requires you to have command over multiple additional concepts like first class objects, higher order functions, closures etc. First, I’ll try to introduce these concepts as necessary and then unravel the core concept of decorator layer by layer. So let’s dive in.

## First Class Objects

In Python, basically everything is an object and functions are regarded as first-class objects. It means that functions can be passed around and used as arguments, just like any other object (string, int, float, list, and so on). You can assign functions to variables and treat them like any other objects. Consider this example:

def func_a():
return "I was angry with my friend."

def func_b():
return "I told my wrath, my wrath did end"

def func_c(*funcs):
for func in funcs:
print(func())

main_func = func_c
main_func(func_a, func_b)

>>> I was angry with my friend.
>>> I told my wrath, my wrath did end


The above example demonstrates how Python treats functions as first class citizens. First, I defined two functions, func_a and func_b and then func_c takes them as parameters. func_c runs the functions taken as parameters and prints the results. Then we assign func_c to variable main_func. Finally, we run main_func and it behaves just like func_c.

## Higher Order Functions

Python also allows you to use functions as return values. You can take in another function and return that function or you can define a function within another function and return the inner function.

def higher(func):
"""This is a higher order function.
It returns another function.
"""

return func

def lower():
return "I'm hunting high and low"

higher(lower)

>>> <function __main__.lower()>


Now you can assign the result of higher to another variable and execute the output function.

h = higher(lower)
h()

>>> "I'm hunting high and low"


Let’s look into another example where you can define a nested function within a function and return the nested function instead of its result.

def outer():
"""Define and return a nested function from another function."""

def inner():
return "Hello from the inner func"

return inner

inn = outer()
inn()

>>> 'Hello from the inner func'


Notice how the nested function inner was defined inside the outer function and then the return statement of the outer function returned the nested function. After definition, to get to the nested function, first we called the outer function and received the result as another function. Then executing the result of the outer function prints out the message from the inner function.

## Closures

You saw examples of inner functions at work in the previous section. Nested functions can access variables of the enclosing scope. In Python, these non-local variables are read only by default and we must declare them explicitly as non-local (using nonlocal keyword) in order to modify them. Following is an example of a nested function accessing a non-local variable.

def burger(name):
def ingredients():
if name == "deli":
return ("steak", "pastrami", "emmental")
elif name == "smashed":
return ("chicken", "nacho cheese", "jalapeno")
else:
return None

return ingredients


Now run the function,

ingr = burger("deli")
ingr()

>>> ('steak', 'pastrami', 'emmental')


Well, that’s unusual.

The burger function was called with the string deli and the returned function was bound to the name ingr. On calling ingr(), the message was still remembered and used to derive the outcome although the outer function burger had already finished its execution.

This technique by which some data (“deli”) gets attached to the code is called closure in Python. The value in the enclosing scope is remembered even when the variable goes out of scope or the function itself is removed from the current namespace. Decorators uses the idea of non-local variables multiple times and soon you’ll see how.

## Writing a Basic Decorator

With these prerequisites out of the way, let’s go ahead and create your first simple decorator.

def deco(func):
def wrapper():
print("This will get printed before the function is called.")
func()
print("This will get printed after the function is called.")

return wrapper


Before using the decorator, let’s define a simple function without any parameters.

def ans():
print(42)


Treating the functions as first-class objects, you can use your decorator like this:

ans = deco(ans)
ans()

>>> This will get printed before the function is called.
42
This will get printed after the function is called.


In the above two lines, you can see a very simple decorator in action. Our deco function takes in a target function, manipulates the target function inside a wrapper function and then returns the wrapper function. Running the function returned by the decorator, you will get your modified result. To put it simply, decorators wraps a function and modifies its behavior.

The decorator function runs at the time the decorated function is imported/defined, not when it is called.

Before moving onto the next section, let’s see how we can get the return value of target function instead of just printing it.

def deco(func):
"""This modified decorator also returns the result of func."""

def wrapper():
print("This will get printed before the function is called.")
ret = func()
print("This will get printed after the function is called.")
return ret

return wrapper

def ans():
return 42


In the above example, the wrapper function returns the result of the target function and the wrapper itself. This makes it possible to get the result of the modified function.

ans = deco(ans)
print(ans())

>>> This will get printed before the function is called.
This will get printed after the function is called.
42


Can you guess why the return value of the decorated function appeared in the last line instead of in the middle like before?

## The @ Syntactic Sugar

The way you’ve used decorator in the last section might feel a little clunky. First, you have to type the name ans three times to call and use the decorator. Also, it becomes harder to tell apart where the decorator is actually working. So Python allows you to use decorator with the special syntax @. You can apply your decorators while defining your functions, like this:

@deco
def func():
...

# Now call your decorated function just like a normal one
func()


Sometimes the above syntax is called the pie syntax and it’s just a syntactic sugar for func = deco(func).

## Decorating Functions with Arguments

The naive decorator that we’ve implemented above will only work for functions that take no arguments. It’ll fail and raise TypeError if your try to decorate a function having arguments with deco. Now let’s create another decorator called yell which will take in a function that returns a string value and transform that string value to uppercase.

def yell(func):
def wrapper(*args, **kwargs):
ret = func(*args, **kwargs)
ret = ret.upper() + "!"
return ret

return wrapper


Create the target function that returns string value.

@yell
def hello(name):
return f"Hello {name}"

hello("redowan")

>>> 'HELLO REDOWAN!'


Function hello takes a name:string as parameter and returns a message as string. Look how the yell decorator is modifying the original return string, transforming that to uppercase and adding an extra ! sign without directly changing any code in the hello function.

## Solving Identity Crisis

In Python, you can introspect any object and its properties via the interactive shell. A function knows its identity, docstring etc. For instance, you can inspect the built in print function in the following ways:

print

>>> <function print>

print.__name__

>>> 'print'

print.__doc__

>>> "print(value, ..., sep=' ', end='\\n', file=sys.stdout, flush=False)\n\nPrints the values to a stream, or to sys.stdout by default.\nOptional keyword arguments:\nfile:  a file-like object (stream); defaults to the current sys.stdout.\nsep:   string inserted between values, default a space.\nend:   string appended after the last value, default a newline.\nflush: whether to forcibly flush the stream."

help(print)

>>> Help on built-in function print in module builtins:

print(...)
print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)

Prints the values to a stream, or to sys.stdout by default.
Optional keyword arguments:
file:  a file-like object (stream); defaults to the current sys.stdout.
sep:   string inserted between values, default a space.
end:   string appended after the last value, default a newline.
flush: whether to forcibly flush the stream.


This introspection works similarly for functions that you defined yourself. I’ll be using the previously defined hello function.

hello.__name__

>>> 'wrapper'

help(hello)

>>> Help on function wrapper in module __main__:
wrapper(*args, **kwargs)


Now what’s going on there. The decorator yell has made the function hello confused about its own identity. Instead of reporting its own name, it takes the identity of the inner function wrapper. This can be confusing while doing debugging. You can fix this by using builtin wraps decorator from the functools module. This will make sure that the original identity of the decorated function stays preserved.

from functools import wraps

def yell(func):
@wraps(func)
def wrapper(*args, **kwargs):
ret = func(*args, **kwargs)
ret = ret.upper() + "!"
return ret

return wrapper

@yell
def hello(name):
"Hello from the other side."
return f"Hello {name}"

hello("Galaxy")

>>> 'HELLO GALAXY!'


Introspecting the hello function decorated with modified decorator will give you the desired result.

hello.__name__

>>> 'hello'

help(hello)

>>> Help on function hello in module __main__:

hello(name)
Hello from the other side.


## Decorators in the Wild

Before moving on to the next section let’s see a few real world examples of decorators. To define all the decorators, we’ll be using the following template that we’ve perfected so far.

from functools import wraps

def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
# Do something before
ret = func(*args, **kwargs)
# Do something after
return ret

return wrapper


### Timer

Timer decorator will help you time your callables in a non-intrusive way. It can help you while debugging and profiling your functions.

from time import perf_counter
from functools import wraps

def timer(func):
"""This decorator prints out the execution time of a callable."""

@wraps(func)
def wrapper(*args, **kwargs):
start_time = perf_counter()
ret = func(*args, **kwargs)
end_time = perf_counter()
run_time = end_time - start_time
print(f"Finished running {func.__name__} in {run_time:.4f} seconds.")
return ret

return wrapper

@timer
def dothings(n_times):
for _ in range(n_times):
return sum((i ** 3 for i in range(100_000)))


In the above way, we can introspect the time it requires for function dothings to complete its execution.

dothings(100_000)

>>> Finished running dothings in 0.0353 seconds.
24999500002500000000


### Exception Logger

Just like the timer decorator, we can define a logger decorator that will log the state of a callable. For this demonstration, I’ll be defining a exception logger that will show additional information like timestamp, argument names when an exception occurs inside of the decorated callable.

from functools import wraps
from datetime import datetime

def logexc(func):
@wraps(func)
def wrapper(*args, **kwargs):

# Stringify the arguments
args_rep = [repr(arg) for arg in args]
kwargs_rep = [f"{k}={v!r}" for k, v in kwargs.items()]
sig = ", ".join(args_rep + kwargs_rep)

# Try running the function
try:
return func(*args, **kwargs)
except Exception as e:
print("Time: ", datetime.now().strftime("%Y-%m-%d [%H:%M:%S]"))
print("Arguments: ", sig)
print("Error:\n")
raise

return wrapper

@logexc
def divint(a, b):
return a / b


Let’s invoke ZeroDivisionError to see the logger in action.

divint(1, 0)

>>> Time:  2020-05-12 [12:03:31]
Arguments:  1, 0
Error:

------------------------------------------------------------

ZeroDivisionError         Traceback (most recent call last)
....


The decorator first prints a few info regarding the function and then raises the original error.

### Validation & Runtime Checks

Python’s type system is strongly typed, but very dynamic. For all its benefits, this means some bugs can try to creep in, which more statically typed languages (like Java) would catch at compile time. Looking beyond even that, you may want to enforce more sophisticated, custom checks on data going in or out. Decorators can let you easily handle all of this, and apply it to many functions at once.

Imagine this: you have a set of functions, each returning a dictionary, which (among other fields) includes a field called “summary.” The value of this summary must not be more than 30 characters long; if violated, that’s an error. Here is a decorator that raises a ValueError if that happens:

from functools import wraps

def validate_summary(func):
@wraps(func)
def wrapper(*args, **kwargs):
ret = func(*args, **kwargs)
if len(ret["summary"]) > 30:
raise ValueError("Summary exceeds 30 character limit.")
return ret

return wrapper

@validate_summary
def short_summary():
return {"summary": "This is a short summary"}

@validate_summary
def long_summary():
return {"summary": "This is a long summary that exceeds character limit."}

print(short_summary())
print(long_summary())

>>> {'summary': 'This is a short summary'}

-------------------------------------------------------------------

ValueError                       Traceback (most recent call last)

<ipython-input-178-7375d8e2a623> in <module>
19
20 print(short_summary())
---> 21 print(long_summary())
...


### Retry

Imagine a situation where your defined callable fails due to some I/O related issues and you’d like to retry that again. Decorator can help you to achieve that in a reusable manner. Let’s define a retry decorator that will rerun the decorated function multiple times if an http error occurs.

from functools import wraps
import requests

def retry(func):
"""This will rerun the decorated callable 3 times if
the callable encounters http 500/404 error."""

@wraps(func)
def wrapper(*args, **kwargs):
n_tries = 3
tries = 0
while True:
resp = func(*args, **kwargs)
if resp.status_code == 500 or resp.status_code == 404 and tries < n_tries:
print(f"retrying... ({tries})")
tries += 1
continue
break
return resp

return wrapper

@retry
def getdata(url):
resp = requests.get(url)
return resp

resp = getdata("https://httpbin.org/get/1")
resp.text

>>> retrying... (0)
retrying... (1)
retrying... (2)

'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>\n'


## Applying Multiple Decorators

You can apply multiple decorators to a function by stacking them on top of each other. Let’s define two simple decorators and use them both on a function.

from functools import wraps

def greet(func):
"""Greet in English."""

@wraps(func)
def wrapper(*args, **kwargs):
ret = func(*args, **kwargs)
return "Hello " + ret + "!"

return wrapper

def flare(func):
"""Add flares to the string."""

@wraps(func)
def wrapper(*args, **kwargs):
ret = func(*args, **kwargs)
return "🎉 " + ret + " 🎉"

return wrapper

@flare
@greet
def getname(name):
return name

getname("Nafi")

>>> '🎉 Hello Nafi! 🎉'


The decorators are called in a bottom up order. First, the decorator greet gets applied on the result of getname function and then the result of greet gets passed to the flare decorator. The decorator stack above can be written as flare(greet(getname(name))). Change the order of the decorators and see what happens!

## Decorators with Arguments

While defining the retry decorator in the previous section, you may have noticed that I’ve hard coded the number of times I’d like the function to retry if an error occurs. It’d be handy if you could inject the number of tries as a parameter into the decorator and make it work accordingly. This is not a trivial task and you’ll need three levels of nested functions to achieve that.

Before doing that let’s cook up a trivial example of how you can define decorators with parameters.

from functools import wraps

def joinby(delimiter=" "):
"""This decorator splits the string output of the
decorated function by a single space and then joins
them using a user specified delimiter."""

def outer_wrapper(func):
@wraps(func)
def inner_wrapper(*args, **kwargs):
ret = func(*args, **kwargs)
ret = ret.split(" ")
ret = delimiter.join(ret)
return ret

return inner_wrapper

return outer_wrapper

@joinby(delimiter=",")
def hello(name):
return f"Hello {name}!"

@joinby(delimiter=">")
def greet(name):
return f"Greetings {name}!"

@joinby()
def goodbye(name):
return f"Goodbye {name}!"

print(hello("Nafi"))
print(greet("Redowan"))
print(goodbye("Delowar"))

>>> Hello,Nafi!
Greetings>Redowan!
Goodbye Delowar!


The decorator joinby takes a single parameter called delimiter. It splits the string output of the decorated function by a single space and then joins them using the user defined delimiter specified in the delimiter argument. The three layer nested definition looks scary but we’ll get to that in a moment. Notice how you can use the decorator with different parameters. In the above example, I’ve defined three different functions to demonstrate the usage of joinby. It’s important to note that in case of a decorator that takes parameters, you’ll always need to pass something to it and even if you don’t want to pass any parameter (run with the default), you’ll still need to decorate your function with deco() instead of deco. Try changing the decorator on the goodbye function from joinby() to joinby and see what happens.

Typically, a decorator creates and returns an inner wrapper function but here in the repeat decorator, there is an inner function within another inner function. This almost looks like a dream within a dream from the movie Inception.

There are a few subtle things happening in the joinby() function:

• Defining outer_wrapper() as an inner function means that repeat() will refer to a function object outer_wrapper.

• The delimiter argument is seemingly not used in joinby() itself. But by passing delimiter a closure is created where the value of delimiter is stored until it will be used later by inner_wrapper()

## Decorators with & without Arguments

You saw earlier that a decorator specifically designed to take parameters can’t be used without parameters; you need to at least apply parenthesis after the decorator deco() to use it without explicitly providing the arguments. But what if you want to design one that can used both with and without arguments. Let’s redefine the joinby decorator so that you can use it with parameters or just like an ordinary parameter-less decorator that we’ve seen before.

from functools import wraps

def joinby(_func=None, *, delimiter=" "):
"""This decorator splits the string output
of a function by a single space and then joins that
using a user specified delimiter."""

def outer_wrapper(func):
@wraps(func)
def inner_wrapper(*args, **kwargs):
ret = func(*args, **kwargs)
ret = ret.split(" ")
ret = delimiter.join(ret)
return ret

return inner_wrapper

# This part enables you to use the decorator with/without arguments
if _func is None:
return outer_wrapper
else:
return outer_wrapper(_func)

@joinby(delimiter=",")
def hello(name):
return f"Hello {name}!"

@joinby
def greet(name):
return f"Greetings {name}!"

print(hello("Nafi"))
print(greet("Redowan"))

>>> Hello,Nafi!
Greetings Redowan!


Here, the _func argument acts as a marker, noting whether the decorator has been called with arguments or not:

If joinby has been called without arguments, the decorated function will be passed in as _func. If it has been called with arguments, then _func will be None. The * in the argument list means that the remaining arguments can’t be called as positional arguments. This time you can use joinby with or without arguments and function hello and greet above demonstrate that.

## A Generic Pattern

Personally, I find it cumbersome how you need three layers of nested functions to define a generalized decorator that can be used with or without arguments. David Beazly in his book Python Cookbook shows an excellent way to define generalized decorators without writing three levels of nested functions. It uses the built in functools.partial function to achieve that. The following is a pattern you can use to define generalized decorators in a more elegant way:

from functools import wraps

def decorator(func=None, foo="spam"):
if func is None:
return functools.partial(decorator, foo=foo)

@wraps(func)
def wrapper(*args, **kwargs):
# Do something with func and foo, if you're so inclined
pass

return wrapper

# Applying decorator without any parameter
@decorator
def f(*args, **kwargs):
pass

# Applying decorator with extra parameter
@decorator(foo="buzz")
def f(*args, **kwargs):
pass


Let’s redefine our retry decorator using this pattern.

from functools import wraps

def retry(func=None, n_tries=4):
if func is None:
return functools.partial(retry, n_tries=n_tries)

@wraps(func)
def wrapper(*args, **kwargs):
tries = 0
while True:
ret = func(*args, **kwargs)
if ret.status_code == 500 or ret.status_code == 404 and tries < n_tries:
print(f"retrying... ({tries})")
tries += 1
continue
break
return ret

return wrapper

@retry
def getdata(url):
resp = requests.get(url)
return resp

@retry(n_tries=2)
def getdata_(url):
resp = requests.get(url)
return resp

resp1 = getdata("https://httpbin.org/get/1")
print("-----------------------")
resp2 = getdata_("https://httpbin.org/get/1")

>>> retrying... (0)
retrying... (1)
retrying... (2)
retrying... (3)
-----------------------
retrying... (0)
retrying... (1)


In this case, you do not have to write three level nested functions and the functools.partial takes care of that. Partials can be used to make new derived functions that have some input parameters pre-assigned.Roughly partial does the following:

def partial(func, *part_args):
def wrapper(*extra_args):
args = list(part_args)
args.extend(extra_args)
return func(*args)

return wrapper


This eliminates the need to write multiple layers of nested factory function get a generalized decorator.

## Defining Decorators with Classes

This time, I’ll be using a class to compose a decorator. Classes can be handy to avoid nested architecture while writing decorators. Also, it can be helpful to use a class while writing stateful decorators. You can follow the pattern below to compose decorators with classes.

from functools import update_wrapper

class ClassDeco:
def __init__(self, func):
update_wrapper(self, func)
self.func = func

def __call__(self, *args, **kwargs):

# You can add some code before the function call
ret = self.func(*args, **kwargs)
# You can also add some code after the function call

return ret


Let’s use the above template to write a decorator named Emphasis that will add bold tags <b></b>to the string output of a function.

from functools import update_wrapper

class Emphasis:
def __init__(self, func):
update_wrapper(self, func)
self.func = func

def __call__(self, *args, **kwargs):
ret = self.func(*args, **kwargs)
return "<b>" + ret + "</b>"

@Emphasis
def hello(name):
return f"Hello {name}"

print(hello("Nafi"))
print(hello("Redowan"))

>>> <b>Hello Nafi</b>
<b>Hello Redowan</b>


The init() method stores a reference to the function num_calls and can do other necessary initialization. The call() method will be called instead of the decorated function. It does essentially the same thing as the wrapper() function in our earlier examples. Note that you need to use the functools.update_wrapper() function instead of @functools.wraps.

Before moving on, let’s write a stateful decorator using classes. Stateful decorators can remember the state of their previous run. Here’s a stateful decorator called Tally that will keep track of the number of times decorated functions are called in a dictionary. The keys of the dictionary will hold the names of the functions and the corresponding values will hold the call count.

from functools import update_wrapper

class Tally:
def __init__(self, func):
update_wrapper(self, func)
self.func = func
self.tally = {}
self.n_calls = 0

def __call__(self, *args, **kwargs):
self.n_calls += 1
self.tally[self.func.__name__] = self.n_calls

print("Callable Tally:", self.tally)
return self.func(*args, **kwargs)

@Tally
def hello(name):
return f"Hello {name}!"

print(hello("Redowan"))
print(hello("Nafi"))

>>> Callable Tally: {'hello': 1}
Hello Redowan!
Callable Tally: {'hello': 2}
Hello Nafi!


## A Few More Examples

### Caching Return Values

Decorators can provide an elegant way of memoizing function return values. Imagine you have an expensive API and you’d like call that as few times as possible. The idea is to save and cache values returned by the API for particular arguments, so that if those arguments appear again, you can serve the results from the cache instead of calling the API again. This can dramatically improve your applications’ performance. Here I’ve simulated an expensive API call and provided caching with a decorator.

import time

def api(a):
"""API takes an integer and returns the square value of it.
To simulate a time consuming process, I've added some time delay to it."""

print("The API has been called...")

# This will delay 3 seconds
time.sleep(3)

return a * a

api(3)

>>> The API has been called...
9


You’ll see that running this function takes roughly 3 seconds. To cache the result , we can use Python’s built in functools.lru_cache to save the result against an argument in a dictionary and serve that when it encounters the same argument again. The only drawback here is, all the arguments need to be hashable.

from functools import lru_cache

@lru_cache(maxsize=32)
def api(a):
"""API takes an integer and returns the square value of it.
To simulate a time consuming process, I've added some time delay to it."""

print("The API has been called...")

# This will delay 3 seconds
time.sleep(3)

return a * a

api(3)

>>> 9


Least Recently Used (LRU) Cache organizes items in order of use, allowing you to quickly identify which item hasn’t been used for the longest amount of time. In the above case, the parameter max_size refers to the maximum numbers of responses to be saved up before it starts deleting the earliest ones. While you run the decorated function, you’ll see first time it’ll take roughly 3 seconds to return the result. But if you rerun the function again with the same parameter it’ll spit the result from the cache almost instantly.

### Unit Conversion

The following decorator converts length from SI units to multiple other units without polluting your target function with conversion logics.

from functools import wraps, partial

def convert(func=None, convert_to=None):
"""This converts value from meter to others."""

if func is None:
return partial(convert, convert_to=convert_to)

@wraps(func)
def wrapper(*args, **kwargs):
print(f"Conversion unit: {convert_to}")

ret = func(*args, **kwargs)

# Adding conversion rules
if convert_to is None:
return ret

elif convert_to == "km":
return ret / 1000

elif convert_to == "mile":
return ret * 0.000621371

elif convert_to == "cm":
return ret * 100

elif convert_to == "mm":
return ret * 1000

else:
raise ValueError("Conversion unit is not supported.")

return wrapper


Let’s use that on a function that returns the area of a rectangle.

@convert(convert_to="mile")
def area(a, b):
return a * b

area(1, 2)

>>> Conversion unit: mile
0.001242742


Using the convert decorator on the area function shows how it prints out the transformation unit before returning the desired result. Experiment with other conversion units and see what happens.

### Function Registration

The following is an example of registering logger function in Flask framework. The decorator register_logger doesn’t make any change to the decorated logger function. Rather it takes the function and registers it in a list called logger_list every time it’s invoked.

from flask import Flask, request

logger_list = []

def register_logger(func):
logger_list.append(func)
return func

def run_loggers(request):
for logger in logger_list:
logger(request)

@register_logger
def logger(request):
print(request.method, request.path)

@app.route("/")
def index():
run_loggers(request)
return "Hello World!"

if __name__ == "__main__":
app.run(host="localhost", port="5000")


If you run the server and hit the http://localhost:5000/ url, it’ll greet you with a Hello World! message. Also you’ll able to see the printed method and path of your http request on the terminal. Moreover, if you inspect the logger_list, you’ll find the registered logger there. You’ll find a lot more real life usage of decorators in the Flask framework.

## Remarks

All the pieces of codes in the blog were written and tested with python 3.8 on a machine running Ubuntu 20.04.