slides/empowering-django-with-background-workers
1
Fork 1
Code Activity Actions
empowering-django-with-back.../slides.md

9.2 KiB
Raw Blame History

title class highlighter transition mdc monaco themeConfig addons
Empowering Django with Background Workers text-center shiki slide-left true false
primary
#0c4b33
slidev-addon-qrcode

Empowering Django with Background Workers

Jake Howard

  • Senior Systems Engineer @ Torchbox
  • Security & Performance @ Wagtail
  • theorangeone.net
  • @RealOrangeOne
  • @RealOrangeOne
  • @jake@theorangeone.net

layout: center

Django isn't just for websites

flowchart LR
    U(User 🧑‍💻)
    D[\Django/]

    U---->|Request|D
    D---->|Response|U

layout: full

flowchart BT
    U[User 🧑‍💻]
    D[\Django/]
    DB[(Database)]
    C[(Cache)]
    E>Email]
    EA[External API]
    V[[Video Transcode]]
    ML((Machine Learning))

    U---->|Request|D
    D---->|Response|U

    D-.-DB & E & EA & V & ML & C

layout: full

flowchart BT
    U[User 🧑‍💻]
    D[\Django/]
    DB[(Database)]
    C[(Cache)]
    E>Email]
    EA[External API]
    V[[Video Transcode]]
    ML((Machine Learning))
    B{{Background Processing}}

    U--->|Request|D
    D--->|Response|U

    D---B

    D-.-C & DB

    B---E & V & ML & EA
    B---C & DB

layout: cover

Background Workers?

layout: fact

flowchart LR
    D[\Django/]
    S[(Queue Store)]
    R1{Runner}
    R2{Runner}
    R3{Runner}

    D<--->S<-..->R1 & R2 & R3

layout: cover

When?

layout: fact

flowchart BT
    D[\Django/]

    subgraph Fast & Reliable
      DB[(Database)]
      C[(Cache)]
    end

    subgraph Slow / Unreliable
      E>Email]
      EA[External API]
      V[[Video Transcode]]
      ML((Machine Learning))
    end

    D---DB & C

    D-.-E & EA & V & ML

layout: cover

Background Workers in Django

layout: cover background: https://docs.celeryq.dev/en/stable/_static/celery_512.png

Celery!

layout: image-right image: https://images.unsplash.com/photo-1444703686981-a3abbc4d4fe3?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D

Others...

  • Celery
  • arq
  • Django DB Queue
  • Django Lightweight Queue
  • Django Too Simple Q
  • Django-Q
  • Django-Q2
  • Dramatiq
  • Huey
  • RQ
  • Taskiq
  • ...

layout: cover background: https://images.unsplash.com/photo-1522096823084-2d1aa8411c13?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D

Email

layout: none

from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

import django_rq

def send_email_to_user(user: User):
    email_content = render_to_string("email-template.html", {"user": user})
    send_mail(
      subject="Here is your important customized message",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

users_to_email = User.objects.all()

for user in users_to_email.iterator():
    django_rq.enqueue(send_email_to_user, user)

layout: center

A problem

```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

import django_rq

def send_email_to_user(user: User):
    email_content = render_to_string("email-template.html", {"user": user})
    send_mail(
      subject="Here is your important customized message",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

users_to_email = User.objects.all()

for user in users_to_email.iterator():
    django_rq.enqueue(send_email_to_user, user)
```

```python {all|5-7,20|all}
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from my_celery_config import app

@app.task
def send_email_to_user(user: User):
    email_content = render_to_string("email-template.html", {"user": user})
    send_mail(
      subject="Here is your important customized message",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

users_to_email = User.objects.all()

for user in users_to_email.iterator():
    send_email_to_user.delay(user)
```

layout: fact

Situation:

There are 14 competing standards.

layout: image-right image: https://images.unsplash.com/photo-1674027444485-cec3da58eef4?q=80&w=1932&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D class: flex items-center text-2xl

  • Application developers
  • Library maintainers
  • System Engineers

layout: image image: /ridiculous.png class: bg-top!

Ridiculous!

layout: fact

Introducing:

django.tasks*

layout: center

```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from my_celery_config import app

@app.task
def send_email_to_user(user: User):
    email_content = render_to_string("email-template.html", {"user": user})
    send_mail(
      subject="Here is your important customized message",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

users_to_email = User.objects.all()

for user in users_to_email.iterator():
    send_email_to_user.delay(user)
```

```python {all|5-7,20|all}
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from django.tasks import task

@task()
def send_email_to_user(user: User):
    email_content = render_to_string("email-template.html", {"user": user})
    send_mail(
      subject="Here is your important customized message",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

users_to_email = User.objects.all()

for user in users_to_email.iterator():
    send_email_to_user.enqueue(user)
```

layout: center

from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

users_to_email = User.objects.all()

email_content = render_to_string("email-template.html")

for user in users_to_email.iterator():
    send_mail(
      subject="Here is your important message",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

# settings.py
EMAIL_BACKEND = "django.core.mail.backends.tasks.SMTPEmailBackend"

layout: image-right image: /soon.png class: flex justify-center text-2xl flex-col

Q: Why something new?

A: We're not

layout: center

Why something built-in?

  • Improve interoperability
  • Reduce cognitive load
  • Reduce barrier to entry

layout: center

{.max-h-32.mx-auto}

vs

{.max-h-48.mx-auto}

layout: section

Where are we now?

pip install django-tasks

layout: section

Where are we going?

layout: image-right image: https://images.unsplash.com/photo-1451187580459-43490279c0fa?q=80&w=1744&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D class: flex justify-center flex-col text-xl

Out of scope

  • Completion / failed hooks
  • Bulk queueing
  • Automated task retrying
  • Task runner API
  • Unified observability
  • Cron-based scheduling
  • Task timeouts
  • Swappable argument serialization

layout: section

What's next?

layout: end

END