658 lines
12 KiB
Markdown
658 lines
12 KiB
Markdown
---
|
|
title: Empowering Django with Background Workers
|
|
class: text-center
|
|
highlighter: shiki
|
|
transition: slide-left
|
|
mdc: true
|
|
monaco: false
|
|
themeConfig:
|
|
primary: '#0c4b33'
|
|
addons:
|
|
- "slidev-addon-qrcode"
|
|
---
|
|
|
|
# Empowering Django with Background Workers
|
|
|
|
### Jake Howard{.mt-10}
|
|
|
|
<ul class="list-none! text-sm [&>li]:m-0!">
|
|
<li>Senior Systems Engineer @ Torchbox <mdi-fire class="fill-white"/></li>
|
|
<li>Core, Security & Performance teams @ Wagtail <logos-wagtail class="fill-white"/></li>
|
|
</ul>
|
|
|
|
<ul class="list-none! text-xs [&>li]:m-0! mt-5">
|
|
<li><mdi-earth /> theorangeone.net</li>
|
|
<li><mdi-twitter /> @RealOrangeOne</li>
|
|
<li><mdi-github /> @RealOrangeOne</li>
|
|
<li><mdi-mastodon /> @jake@theorangeone.net</li>
|
|
</ul>
|
|
|
|
<div class="absolute right-0 bottom-4">
|
|
<QRCode
|
|
:width="110"
|
|
:height="110"
|
|
data="https://0rng.one/dceu24"
|
|
:dotsOptions="{ color: 'white' }"
|
|
/>
|
|
</div>
|
|
|
|
---
|
|
layout: center
|
|
---
|
|
|
|
# Django is a web framework
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
U(User 🧑💻)
|
|
D[\Django/]
|
|
|
|
U---->|Request|D
|
|
D---->|Response|U
|
|
```
|
|
|
|
<style>
|
|
.mermaid {
|
|
text-align: center;
|
|
}
|
|
</style>
|
|
|
|
---
|
|
layout: full
|
|
---
|
|
|
|
# Django isn't _just_ for websites
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
U[User 🧑💻]
|
|
D[\Django/]
|
|
DB[(Database)]
|
|
E>Email]
|
|
EA[External API]
|
|
V[[Video Transcoding]]
|
|
R[Reporting]
|
|
ML((Machine<br>Learning))
|
|
|
|
U<--->D
|
|
|
|
D---DB
|
|
D-..-E & EA & V & R & ML
|
|
```
|
|
|
|
<style>
|
|
.mermaid {
|
|
text-align: center;
|
|
}
|
|
</style>
|
|
|
|
|
|
---
|
|
layout: full
|
|
---
|
|
|
|
# Background Workers!
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
U[User 🧑💻]
|
|
D[\Django/]
|
|
|
|
E>Email]
|
|
EA[External API]
|
|
V[[Video Transcoding]]
|
|
R[Reporting]
|
|
ML((Machine<br>Learning))
|
|
|
|
B{{<strong>Background Processing</strong>}}
|
|
|
|
U<-->D
|
|
|
|
D-..-B
|
|
|
|
B---E & EA & V & R & ML
|
|
```
|
|
|
|
<style>
|
|
.mermaid {
|
|
text-align: center;
|
|
}
|
|
</style>
|
|
|
|
---
|
|
layout: section
|
|
---
|
|
|
|
# Background Workers?
|
|
|
|
---
|
|
layout: fact
|
|
---
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
D[\Django/]
|
|
S[(Queue Store)]
|
|
R1{Runner}
|
|
R2{Runner}
|
|
R3{Runner}
|
|
|
|
D<----->S<-....->R1 & R2 & R3
|
|
```
|
|
|
|
---
|
|
layout: section
|
|
---
|
|
|
|
# When?
|
|
|
|
---
|
|
layout: cover
|
|
background: https://images.unsplash.com/photo-1518729371765-043e54eb5674?q=80&w=1807&auto=format&fit=crop&ixlib=rb-4.0.3
|
|
---
|
|
|
|
# Does it take time?{.text-right}
|
|
|
|
---
|
|
layout: fact
|
|
---
|
|
|
|
## Does it leave your infrastructure?{.mb-5}
|
|
|
|
```mermaid
|
|
flowchart BT
|
|
D[\Django/]
|
|
|
|
subgraph Slow / Unreliable
|
|
E>Email]
|
|
EA[External API]
|
|
V[[Video Transcode]]
|
|
R[Reporting]
|
|
ML((Machine<br>Learning))
|
|
end
|
|
|
|
subgraph Fast & Reliable
|
|
DB[(Database)]
|
|
C[(Cache)]
|
|
end
|
|
|
|
D---DB & C
|
|
|
|
D-.-E & EA & V & R & ML
|
|
```
|
|
|
|
---
|
|
layout: cover
|
|
background: https://images.unsplash.com/photo-1518770660439-4636190af475?q=80&w=3870&auto=format&fit=crop&ixlib=rb-4.0.3
|
|
---
|
|
|
|
# Specialized hardware?
|
|
|
|
---
|
|
layout: cover
|
|
background: https://images.unsplash.com/photo-1711606815631-38d32cdaec3e?q=80&w=2070&auto=format&fit=crop&ixlib=rb-4.0.3
|
|
---
|
|
|
|
## Example:
|
|
# Complex reporting
|
|
|
|
---
|
|
layout: section
|
|
---
|
|
|
|
# Background Workers in
|
|
|
|
<logos-django class="[&>path]:fill-white! h-fit w-60 -mt-20"/>
|
|
|
|
---
|
|
layout: cover
|
|
background: /celery.svg
|
|
---
|
|
|
|
# Celery!
|
|
|
|
<style>
|
|
.slidev-layout {
|
|
background: white;
|
|
background-size: contain !important;
|
|
}
|
|
</style>
|
|
|
|
---
|
|
layout: image-right
|
|
image: https://images.unsplash.com/photo-1444703686981-a3abbc4d4fe3?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3
|
|
---
|
|
|
|
# Others...
|
|
|
|
<Transform :scale="1.05">
|
|
|
|
- ~~Celery~~
|
|
- arq
|
|
- Django DB Queue
|
|
- Django Lightweight Queue
|
|
- Django Too Simple Q
|
|
- Django-Q
|
|
- Django-Q2
|
|
- Dramatiq
|
|
- Huey
|
|
- RQ
|
|
- Taskiq
|
|
- ...
|
|
|
|
</Transform>
|
|
|
|
---
|
|
layout: cover
|
|
background: https://images.unsplash.com/photo-1522096823084-2d1aa8411c13?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3
|
|
---
|
|
|
|
## Example:
|
|
# Email <mdi-email-fast-outline />
|
|
|
|
---
|
|
layout: center
|
|
---
|
|
|
|
# Sending an email
|
|
|
|
```python {all|7|8|9-14|all}
|
|
from django.contrib.auth.models import User
|
|
from django.core.mail import send_mail
|
|
from django.template.loader import render_to_string
|
|
|
|
from wagtail.models import Page
|
|
|
|
for user in page.subscribers.iterator():
|
|
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
|
|
send_mail(
|
|
subject=f"A change to {page.title} has been published",
|
|
message=email_content
|
|
from_email=None, # Use the default sender email
|
|
recipient_list=[user.email]
|
|
)
|
|
```
|
|
|
|
---
|
|
layout: center
|
|
---
|
|
|
|
```python {all|18|19|10|11-16|all|18-19|all}
|
|
from django.contrib.auth.models import User
|
|
from django.core.mail import send_mail
|
|
from django.template.loader import render_to_string
|
|
|
|
import django_rq
|
|
|
|
from wagtail.models import Page
|
|
|
|
def send_email_to_user(page: Page, user: User):
|
|
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
|
|
send_mail(
|
|
subject=f"A change to {page.title} has been published",
|
|
message=email_content
|
|
from_email=None, # Use the default sender email
|
|
recipient_list=[user.email]
|
|
)
|
|
|
|
for user in page.subscribers.iterator():
|
|
django_rq.enqueue(send_email_to_user, user)
|
|
```
|
|
|
|
---
|
|
layout: center
|
|
---
|
|
|
|
# Using <span v-click.hide="1">RQ</span><span v-click="1"><s class="opacity-60">RQ</s> Celery</span>
|
|
|
|
````md magic-move
|
|
```python
|
|
from django.contrib.auth.models import User
|
|
from django.core.mail import send_mail
|
|
from django.template.loader import render_to_string
|
|
|
|
import django_rq
|
|
|
|
from wagtail.models import Page
|
|
|
|
def send_email_to_user(page: Page, user: User):
|
|
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
|
|
send_mail(
|
|
subject=f"A change to {page.title} has been published",
|
|
message=email_content
|
|
from_email=None, # Use the default sender email
|
|
recipient_list=[user.email]
|
|
)
|
|
|
|
for user in page.subscribers.iterator():
|
|
django_rq.enqueue(send_email_to_user, user)
|
|
```
|
|
|
|
```python {all|7-9,20|all}
|
|
from django.contrib.auth.models import User
|
|
from django.core.mail import send_mail
|
|
from django.template.loader import render_to_string
|
|
|
|
from wagtail.models import Page
|
|
|
|
from my_celery_config import app
|
|
|
|
@app.task
|
|
def send_email_to_user(page: Page, user: User):
|
|
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
|
|
send_mail(
|
|
subject=f"A change to {page.title} has been published",
|
|
message=email_content
|
|
from_email=None, # Use the default sender email
|
|
recipient_list=[user.email]
|
|
)
|
|
|
|
for user in page.subscribers.iterator():
|
|
send_email_to_user.delay(user)
|
|
```
|
|
````
|
|
|
|
<style>
|
|
.slidev-vclick-hidden {
|
|
display: none;
|
|
}
|
|
</style>
|
|
|
|
---
|
|
layout: image
|
|
image: /situation.png
|
|
backgroundSize: 50%
|
|
---
|
|
|
|
---
|
|
layout: image
|
|
image: /ridiculous.png
|
|
backgroundSize: 49%
|
|
---
|
|
|
|
---
|
|
layout: fact
|
|
---
|
|
|
|
## Introducing*:{.mb-5}
|
|
|
|
# `django.tasks`
|
|
|
|
<div class="absolute right-1/2 translate-x-1/2 mt-12">
|
|
<QRCode
|
|
:width="120"
|
|
:height="120"
|
|
data="https://pypi.org/project/django-tasks/"
|
|
:dotsOptions="{ color: 'white' }"
|
|
/>
|
|
</div>
|
|
|
|
---
|
|
layout: image-right
|
|
image: https://images.unsplash.com/photo-1674027444485-cec3da58eef4?q=80&w=1932&auto=format&fit=crop&ixlib=rb-4.0.3
|
|
class: flex items-center text-xl
|
|
---
|
|
|
|
- API contract between library and application developers
|
|
- Swappable backends through `settings.py`
|
|
- Built in backends:
|
|
- ORM
|
|
- "Immediate"
|
|
- "Dummy"
|
|
- Django 5.2 🤞
|
|
- Backport for 4.2+
|
|
---
|
|
layout: center
|
|
---
|
|
|
|
# <span v-click.hide="1">Using Celery</span><span v-click="1">Using <code>django.tasks</code></span>
|
|
|
|
````md magic-move
|
|
```python
|
|
from django.contrib.auth.models import User
|
|
from django.core.mail import send_mail
|
|
from django.template.loader import render_to_string
|
|
|
|
from wagtail.models import Page
|
|
|
|
from my_celery_config import app
|
|
|
|
@app.task
|
|
def send_email_to_user(page: Page, user: User):
|
|
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
|
|
send_mail(
|
|
subject=f"A change to {page.title} has been published",
|
|
message=email_content
|
|
from_email=None, # Use the default sender email
|
|
recipient_list=[user.email]
|
|
)
|
|
|
|
for user in page.subscribers.iterator():
|
|
send_email_to_user.delay(user)
|
|
```
|
|
|
|
```python {all|7-9,20|all}
|
|
from django.contrib.auth.models import User
|
|
from django.core.mail import send_mail
|
|
from django.template.loader import render_to_string
|
|
|
|
from wagtail.models import Page
|
|
|
|
from django.tasks import task
|
|
|
|
@task()
|
|
def send_email_to_user(page: Page, user: User):
|
|
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
|
|
send_mail(
|
|
subject=f"A change to {page.title} has been published",
|
|
message=email_content
|
|
from_email=None, # Use the default sender email
|
|
recipient_list=[user.email]
|
|
)
|
|
|
|
for user in page.subscribers.iterator():
|
|
send_email_to_user.enqueue(user)
|
|
```
|
|
````
|
|
|
|
<style>
|
|
.slidev-vclick-hidden {
|
|
display: none;
|
|
}
|
|
</style>
|
|
|
|
---
|
|
layout: center
|
|
---
|
|
|
|
```python
|
|
from django.contrib.auth.models import User
|
|
from django.core.mail import send_mail
|
|
from django.template.loader import render_to_string
|
|
|
|
from wagtail.models import Page
|
|
|
|
for user in page.subscribers.iterator():
|
|
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
|
|
send_mail(
|
|
subject=f"A change to {page.title} has been published",
|
|
message=email_content
|
|
from_email=None, # Use the default sender email
|
|
recipient_list=[user.email]
|
|
)
|
|
```
|
|
|
|
<br />
|
|
|
|
<v-click>
|
|
|
|
```python
|
|
# settings.py
|
|
EMAIL_BACKEND = "django.core.mail.backends.tasks.SMTPEmailBackend"
|
|
```
|
|
|
|
</v-click>
|
|
|
|
---
|
|
layout: image-right
|
|
image: /soon.png
|
|
class: flex justify-center text-2xl flex-col
|
|
---
|
|
|
|
# Q: Why something new?
|
|
|
|
---
|
|
layout: image-right
|
|
image: https://images.unsplash.com/photo-1525683879097-8babce1c602a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3
|
|
class: flex justify-center text-xl flex-col
|
|
---
|
|
|
|
# Q: Why something built-in?
|
|
|
|
- Reduce barrier to entry
|
|
- Reduce cognitive load
|
|
- Reduce complexity for smaller projects
|
|
- Improve interoperability
|
|
- Use what's already there
|
|
- A common API
|
|
|
|
---
|
|
layout: center
|
|
transition: fade
|
|
---
|
|
|
|
{.h-32.mx-auto}
|
|
|
|
## vs
|
|
|
|
{.h-36.mx-auto}
|
|
|
|
|
|
<style>
|
|
.slidev-layout {
|
|
background: white;
|
|
color: black;
|
|
text-align: center;
|
|
}
|
|
</style>
|
|
|
|
---
|
|
layout: center
|
|
---
|
|
|
|
{.h-32.mx-auto}
|
|
|
|
## vs
|
|
|
|
{.max-h-36.mx-auto}
|
|
|
|
|
|
<style>
|
|
.slidev-layout {
|
|
background: white;
|
|
color: black;
|
|
text-align: center;
|
|
}
|
|
</style>
|
|
|
|
---
|
|
layout: section
|
|
---
|
|
|
|
# Where are we now?
|
|
|
|
---
|
|
layout: image
|
|
image: /dep.png
|
|
---
|
|
|
|
---
|
|
layout: cover
|
|
---
|
|
|
|
# `pip install django-tasks`
|
|
|
|
<div class="absolute right-1/2 translate-x-1/2 mt-12">
|
|
<QRCode
|
|
:width="120"
|
|
:height="120"
|
|
data="https://pypi.org/project/django-tasks/"
|
|
:dotsOptions="{ color: 'white' }"
|
|
/>
|
|
</div>
|
|
|
|
---
|
|
layout: section
|
|
---
|
|
|
|
# Where will we be _soon_™?
|
|
|
|
---
|
|
layout: cover
|
|
background: /celery.svg
|
|
---
|
|
|
|
# Is this the end?
|
|
|
|
<style>
|
|
.slidev-layout {
|
|
background: white;
|
|
background-size: contain !important;
|
|
}
|
|
</style>
|
|
|
|
|
|
---
|
|
layout: image-right
|
|
image: https://images.unsplash.com/photo-1451187580459-43490279c0fa?q=80&w=1744&auto=format&fit=crop&ixlib=rb-4.0.3
|
|
class: flex justify-center flex-col text-xl
|
|
---
|
|
|
|
# Out of scope
|
|
|
|
- Completion / failed hooks
|
|
- Bulk queueing
|
|
- Automated task retrying
|
|
- Task runner API
|
|
- Unified observability
|
|
- Cron-based scheduling
|
|
- Task timeouts
|
|
- Swappable argument serialization
|
|
- ...
|
|
|
|
---
|
|
layout: cover
|
|
background: https://images.unsplash.com/photo-1519187903022-c0055ec4036a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3
|
|
---
|
|
|
|
# The future is bright
|
|
|
|
---
|
|
layout: section
|
|
---
|
|
|
|
# What's next?
|
|
|
|
<div class="absolute right-1/2 translate-x-1/2 mt-12">
|
|
<QRCode
|
|
:width="120"
|
|
:height="120"
|
|
data="https://pypi.org/project/django-tasks/"
|
|
:dotsOptions="{ color: 'white' }"
|
|
/>
|
|
</div>
|
|
|
|
---
|
|
layout: end
|
|
---
|
|
|
|
END
|
|
|
|
<div class="absolute right-1/2 bottom-4 translate-x-1/2">
|
|
<QRCode
|
|
:width="110"
|
|
:height="110"
|
|
data="https://0rng.one/dceu24"
|
|
:dotsOptions="{ color: 'white' }"
|
|
/>
|
|
</div>
|