Back in the days when I was working as a data analyst, I used to spend hours inside Jupyter notebooks exploring, wrangling, and plotting data to gain insights. However, as I shifted my career gear towards backend software development, my usage of interactive exploratory tools dwindled.
Nowadays, I spend the majority of my time working on a fairly large Django monolith accompanied by a fleet of microservices. Although I love my text editor and terminal emulators, I miss the ability to just start a Jupyter Notebook server and run code snippets interactively. While Django allows you to open up a shell environment and run code snippets interactively, it still isn't as flexible as a notebook.
So, I wanted to see if I could connect a Jupyter notebook server to a containerized Django application running on my local machine and interactively start making queries from there. Turns out, you can do that by integrating three tools into your Dockerized environment: ipykernel, jupyter, and django-extensions. Before I start explaining how everything is tied together, here's a fully working example of a containerized Django application where you can log into the Jupyter server and start debugging the app.
The app is just a Dockerized version of the famous polls-app from the Django tutorial.
The directory structure looks as follows:
../django-jupyter/
├── Dockerfile
├── docker-compose.yml
├── mysite
│ ├── db.sqlite3
│ ├── manage.py
│ ├── mysite
│ │ ├── __init__.py
│ │ ├── _debug_settings.py
│ │ ├── asgi.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
│ ├── polls
│ │ ├── __init__.py
│ │ ├── admin.py
│ │ ├── apps.py
│ │ ├── migrations
│ │ │ ├── 0001_initial.py
│ │ │ └── __init__.py
│ │ ├── models.py
│ │ ├── tests.py
│ │ ├── urls.py
│ │ └── views.py
│ └── script.ipynb
├── requirements.txt
└── requirements-dev.txt
We define and pin the dependencies required for the Jupyter integration in the
requirements-dev.txt file:
# These pinned deps will probably get outdated by the time you're reading it.
# Use the latest version but always pin them in applications.
ipykernel==6.20.1
jupyter==1.0.0
django-extensions==3.2.1
The application dependencies are defined in the requirements.txt file:
django==4.1.5
In the mysite/mysite/_debug_settings.py file, we import the configs from the primary
settings file and add the Jupyter configuration attributes there. Here's the full
content of the extended _debug_settings.py file:
from .settings import * # noqa
INSTALLED_APPS.append("django_extensions") # noqa
SHELL_PLUS = "ipython"
SHELL_PLUS_PRINT_SQL = True
IPYTHON_ARGUMENTS = [
"--ext",
"django_extensions.management.notebook_extension",
"--debug",
]
IPYTHON_KERNEL_DISPLAY_NAME = "Django Shell-Plus"
NOTEBOOK_ARGUMENTS = [
"--ip",
"0.0.0.0",
"--port",
"8895",
"--allow-root",
"--no-browser",
"--NotebookApp.iopub_data_rate_limit=1e5",
"--NotebookApp.token=''",
]
DJANGO_ALLOW_ASYNC_UNSAFE = True
Notice how we're appending the django_extensions app to the INSTALLED_APPS list
defined in the main settings file. Then we're setting the shell to ipython with the
SHELL_PLUS attribute. The NOTEBOOK_ARGUMENTS defines the port of the Jupyter server
and some auth-specific settings.
Next, in the Dockerfile, we're defining the application like this:
# Dockerfile
FROM python:3.11-bullseye
# Set the working directory inside the container.
WORKDIR /code
# Don't write .pyc files and make the output unbuffered.
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Install dependencies.
RUN pip install --upgrade pip
COPY requirements.txt requirements-dev.txt ./
RUN pip install -r requirements.txt -r requirements-dev.txt
# Copy the project code.
COPY . /code
Finally, we're orchestrating the application and the Jupyter server in the
docker-compose.yml file. Here's how it looks:
version: "3.9"
services:
web:
build: .
working_dir: /code/mysite
volumes:
- .:/code
webserver:
extends:
service: web
command: python manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
jupyter:
extends:
service: web
environment:
- DJANGO_SETTINGS_MODULE=mysite._debug_settings
- DJANGO_ALLOW_ASYNC_UNSAFE=true
command:
python manage.py shell_plus --notebook
ports:
- "8895:8895"
debug:
extends:
service: web
working_dir: /code
command: sleep infinity
We're orchestrating three services here: webserver, jupyter, and debug. All of
them extend the base web service that builds the Dockerfile. The webserver service
is where the Django app is run and exposed via the 8000 port. The jupyter service
runs the Jupyter server and makes it accessible through your browser via the 8895
port. Additionally, note how we are using our extended version of the main settings by
overriding the DJANGO_SETTINGS_MODULE environment variable and setting it to
mysite._debug_settings. The debug container is spun up to run the migration commands
and perform other maintenance tasks within the container network. All the maintenance
commands are defined in the Makefile for your convenience. You can run any of these by
running make <target> from the root directory.
And that's it!
If you have Docker and docker-compose installed on your local system, you can give it a try. Clone the example-app repo, navigate to the root directory and run:
docker compose up -d
Then run the migration command:
docker compose exec debug python mysite/manage.py makemigrations \
&& docker compose exec debug python mysite/manage.py migrate
Now head over to your browser and go to http://localhost:8000. You should see an empty
page with a simple header like this:
If you go to http://localhost:8895, you'll be able to open a new notebook that
automatically connects to your database and allows you to write interactive code
immediately.
You can run the following snippet and it'll create two questions and two choices in the database.
from polls import models as polls_models
from datetime import datetime, timezone
for question_text in ("Are you okay?", "Do you wanna go there?"):
question = polls_models.Question.objects.create(
question_text=question_text, pub_date=datetime.now(tz=timezone.utc)
)
question.choice_set.set(
polls_models.Choice.objects.create(choice_text=ctext)
for ctext in ("yes", "no")
)
If you run this and refresh your application server, you'll that the objects have been created and they appear in the view:
