27 KiB
27 KiB
title | class | highlighter | transition | mdc | monaco | themeConfig | ||
---|---|---|---|---|---|---|---|---|
Empowering Django with Background Workers | text-center | shiki | slide-left | true | false |
|
Empowering with Background Workers
Jake Howard
Djangocon Europe 2024
layout: full
$ whoami
- Senior Systems Engineer @ Torchbox
- Core, Security & Performance teams @ Wagtail
- I'm an avid self-hoster
- I help students build robots in my "spare" time (https://srobo.org)
- theorangeone.net
- @RealOrangeOne
- @RealOrangeOne
- @jake@theorangeone.net
![](/dceu24-qrcode.png)
layout: center
Django is a web framework
flowchart LR
U(User 🧑💻)
D[\Django/]
U---->|Request|D
D---->|Response|U
layout: full
Django isn't just for websites
flowchart BT
U[User 🧑💻]
D[\Django/]
DB[(Database)]
E>Email]
EA[External API]
V[[Video Transcoding]]
R[Reporting]
ML((Machine<br>Learning))
U<--->D
D---DB
D-..-E & EA & V & R & ML
layout: full
Background Workers?
flowchart BT
U[User 🧑💻]
D[\Django/]
E>Email]
EA[External API]
V[[Video Transcoding]]
R[Reporting]
ML((Machine<br>Learning))
B{{<strong>Background Worker</strong>}}
U<-->D
D-..-B
B---E & EA & V & R & ML
layout: section
Background worker architecture
flowchart LR
D[\Django/]
S[(Queue Store)]
R1{Runner}
R2{Runner}
R3{Runner}
D<----->S<-....->R1 & R2 & R3
layout: section
When?
layout: cover background: https://images.unsplash.com/photo-1518729371765-043e54eb5674?q=80&w=1807&auto=format&fit=crop&ixlib=rb-4.0.3
Does it take time?
layout: fact
Does it leave your infrastructure?
flowchart BT
D[\Django/]
subgraph Slow / Unreliable
E>Email]
EA[External API]
V[[Video Transcode]]
R[Reporting]
ML((Machine<br>Learning))
end
subgraph Fast & Reliable
DB[(Database)]
C[(Cache)]
end
D---DB & C
D-.-E & EA & V & R & ML
layout: cover background: https://images.unsplash.com/photo-1518770660439-4636190af475?q=80&w=3870&auto=format&fit=crop&ixlib=rb-4.0.3
Specialized hardware?
layout: image-right image: https://images.unsplash.com/photo-1711606815631-38d32cdaec3e?q=80&w=2070&auto=format&fit=crop&ixlib=rb-4.0.3 class: text-xl
Examples
- Compiling code
- Complex reporting
- File uploads
- Model training
- PDF generation
- Resizing images
- Sending email
- Transcoding video
- ... 🤯
layout: section
Background Workers in
layout: image-right image: https://images.unsplash.com/photo-1444703686981-a3abbc4d4fe3?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3
Libraries
- Celery
- arq
- Django DB Queue
- Django Lightweight Queue
- Django Too Simple Q
- Django-Q
- Django-Q2
- Dramatiq
- Huey
- RQ
- Taskiq
- ...
layout: cover background: https://images.unsplash.com/photo-1522096823084-2d1aa8411c13?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3
Example:
layout: center
Sending an email
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
from wagtail.models import Page
for user in page.subscribers.iterator():
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
send_mail(
subject=f"A change to {page.title} has been published",
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
layout: center
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
import django_rq
from wagtail.models import Page
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
send_mail(
subject=f"A change to {page.title} has been published",
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
for user in page.subscribers.iterator():
django_rq.enqueue(send_email_to_user, user)
layout: center
Using RQRQ Celery
```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
import django_rq
from wagtail.models import Page
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
send_mail(
subject=f"A change to {page.title} has been published",
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
for user in page.subscribers.iterator():
django_rq.enqueue(send_email_to_user, user)
```
```python {all|7-9,20|all}
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
from wagtail.models import Page
from my_celery_config import app
@app.task
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
send_mail(
subject=f"A change to {page.title} has been published",
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
for user in page.subscribers.iterator():
send_email_to_user.delay(user)
```
layout: image image: /situation.png backgroundSize: 50%
layout: image image: /ridiculous.png backgroundSize: 49%
layout: fact
Introducing*:
django.tasks
![](/django-tasks-qrcode.png)
layout: image-right image: https://images.unsplash.com/photo-1674027444485-cec3da58eef4?q=80&w=1932&auto=format&fit=crop&ixlib=rb-4.0.3 class: flex items-center text-xl
- API contract between library and application developers
- Swappable backends through
settings.py
- Built in implementations:
- ORM
- "Immediate"
- "Dummy"
- Django 5.2 🤞
- Backport for 4.2+
layout: center
Using CeleryUsing django.tasks
```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
from wagtail.models import Page
from my_celery_config import app
@app.task
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
send_mail(
subject=f"A change to {page.title} has been published",
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
for user in page.subscribers.iterator():
send_email_to_user.delay(user)
```
```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
from wagtail.models import Page
from django.tasks import task
@task()
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
send_mail(
subject=f"A change to {page.title} has been published",
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
for user in page.subscribers.iterator():
send_email_to_user.enqueue(user)
```
layout: center
# settings.py
EMAIL_BACKEND = "django.core.mail.backends.tasks.SMTPEmailBackend"
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
from wagtail.models import Page
for user in page.subscribers.iterator():
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
send_mail(
subject=f"A change to {page.title} has been published",
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
layout: image-right image: /soon.png class: flex justify-center text-2xl flex-col
Q: Why something new?
layout: image-right image: https://images.unsplash.com/photo-1525683879097-8babce1c602a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3 class: flex justify-center text-xl flex-col
Q: Why something built-in?
- Reduce barrier to entry
- Reduce cognitive load
- Reduce complexity for smaller projects
- Improve interoperability
- Use what's already there
- A common API
layout: center transition: fade
vs
layout: center
vs
layout: section
Where are we now?
layout: image image: /dep.png
layout: section
pip install django-tasks
![](/django-tasks-qrcode.png)
layout: section
Where will we be soon™?
layout: cover background: /celery.svg
Is this the end?
layout: image-right image: https://images.unsplash.com/photo-1451187580459-43490279c0fa?q=80&w=1744&auto=format&fit=crop&ixlib=rb-4.0.3 class: flex justify-center flex-col text-xl
Out of scope (for now)
- Completion / failed hooks
- Bulk queueing
- Automated task retrying
- Task runner API
- Unified observability
- Cron-based scheduling
- Task timeouts
- Swappable argument serialization
- ...
layout: cover background: https://images.unsplash.com/photo-1519187903022-c0055ec4036a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3
The future is bright
layout: section
What's next?
pip install django-tasks
![](/django-tasks-qrcode.png)
layout: section class: text-center text-2xl
Let's chat!
- theorangeone.net
- @RealOrangeOne
- @RealOrangeOne
- @jake@theorangeone.net
Me
![](/dceu24-qrcode.png)
django-tasks
![](/django-tasks-qrcode.png)
layout: end
END