Toggle Menu

Insights > Tech Tips > Scaling Django Channels with Docker

Scaling Django Channels with Docker

Earlier this summer, I attended PyCon in Portland, Oregon. The talk that excited me the most, by far, was Andrew Godwin’s introduction to Django Channels. In a nutshell, Django Channels creates a simple mechanism for writing web applications in Django that support the HTTP2 Websockets protocol. Websockets are an exciting way to implement asynchronous content, […]

By

August 02, 2016

Earlier this summer, I attended PyCon in Portland, Oregon. The talk that excited me the most, by far, was Andrew Godwin’s introduction to Django Channels. In a nutshell, Django Channels creates a simple mechanism for writing web applications in Django that support the HTTP2 Websockets protocol. Websockets are an exciting way to implement asynchronous content, like chat rooms and live feeds, where updates can be pushed without the need for polling.

Django Channels is fairly easy to get running, especially if you reference Andrew Godwin’s sample applications. One thing Andrew’s examples don’t highlight, however, is that the Channels workers are incredibly easy to scale.

Another Django Channels tutorial, written by Django core developer Jacob Kaplan-Moss, covers how to deploy and scale a Django Channels application in Heroku. Part of his guide addresses the need to scale Channels workers as load increases. Workers essentially take Channels tasks off the Redis queue and executes them, similar to Celery. I suggest you read his guide, it’s fantastic, but the excerpt below highlights how simple it is to scale the Channels worker process in Heroku:

[pcsh lang=”python” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

heroku ps:scale worker=3

[/pcsh]

Easy right? Of course, not everyone uses Heroku for deploying web applications. If you’re using another cloud service or hosting your own infrastructure, technically all you need to do is run the following command on several different servers:

[pcsh lang=”python” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

./manage.py runworker 

[/pcsh]

The logistics of managing the worker processes is not trivial using the solution above, especially if you want your worker capacity to be dynamic based on the current or predicted load.

Splitting and scaling the web interface and worker processes is a great use case for containerizing with Docker and Docker Compose. Let’s go back to Andrew Godwin’s channels-examples repo, where you’ll find docker-compose.yml configurations for running his samples. If you look closely, you’ll notice that the configuration defines a single Django container that executes {code}python manage.py runserver{code}, which is not suitable for production, nor is it scalable.

In the following example, I’ll be updating Andrew’s multichat app to make it scalable. Follow the instructions in Andrew’s README, but replace the default docker-compose.yml with the following before running any docker-compose commands:

New docker-compose.yml:

[pcsh lang=”python” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

version: "2"

services:
 redis:
  image: redis:latest
 web:
  build: .
  command: daphne -b 0.0.0.0 -p 8000 multichat.asgi:channel_layer
  volumes:
   - .:/code
  ports:
   - "8000:8000"
  links:
   - redis
 worker:
  build: .
  command: python manage.py runworker
  volumes:
   - .:/code
  links:
   - redis

[/pcsh]

Running `./manage.py runserver`, as Andrew’s original docker-compose.yml did, will create a single process with two threads: the core Django interface and a Channels worker process. Since we want to scale the worker processes, the new docker-compose.yml breaks the two threads into separate containers: ‘web’ and ‘worker’.

The web container no longer uses runserver, but instead uses daphne, which is a web server like Gunicorn, but supports ASGI, a Websockets-compatible derivative of WSGI. Daphne is only serving the core Django interface, so the worker process is expected to run separately. This is essentially the same stack created by Jacob in his Heroku guide.

With the new docker-compose.yml in place, running {code}docker-compose up -d{code} creates three containers: Redis, the Django web server, and the Channels worker process. Channels tasks are received by the web server, placed in a Redis queue, and then consumed by the worker process.

[pcsh lang=”python” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

~/channels-examples/multichat# docker-compose ps
    Name           Command        State      Ports
------------------------------------------------------------------------------------
multichat_redis_1  docker-entrypoint.sh redis ...  Up   6379/tcp
multichat_web_1   daphne -b 0.0.0.0 -p 8000 ...  Up   0.0.0.0:8000->8000/tcp
multichat_worker_1  python manage.py runworker    Up

[/pcsh]

At this point, scaling the workers is just as easy as Heroku:

[pcsh lang=”python” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

docker-compose scale worker=3

[/pcsh]

Listing the processes show two new workers:

[pcsh lang=”python” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]

~/channels-examples/multichat# docker-compose ps
    Name           Command        State      Ports
------------------------------------------------------------------------------------
multichat_redis_1  docker-entrypoint.sh redis ...  Up   6379/tcp
multichat_web_1   daphne -b 0.0.0.0 -p 8000 ...  Up   0.0.0.0:8000->8000/tcp
multichat_worker_1  python manage.py runworker    Up
multichat_worker_2  python manage.py runworker    Up
multichat_worker_3  python manage.py runworker    Up

[/pcsh]

Scaling the Redis and web containers is also possible, but since they bind to static ports they require a load balancer to scale properly. This is not difficult, but is beyond the scope of this guide. The worker processes don’t need a load balancer because the processes are simple consumers and are not binding to any ports.

For a production deployment, you’d probably want to plug Compose into a Swarm cluster so that the worker processes actually run on separate machines. Other tools exist for managing clusters, such as Mesos and Kubernetes. Regardless of the tool you use, the concept is simple: once workers are containerized, scaling is as easy as creating more instances.

You Might Also Like

Events

Best Tech Events for September 2018

Summer is coming to a close, but we still have events for you to attend!...

Tech Tips

Creating a RESTful API: Django REST Framework vs. Flask

There are a multitude of frameworks for creating RESTful APIs in just about every popular...

Tech Tips

How Do I Internationalize My Django App?

Internationalizing a website is hard. If you have a simple site and are working with...