bringing-background-workers.../slides.md

623 lines
11 KiB
Markdown
Raw Normal View History

2024-04-19 11:31:54 +01:00
---
title: Empowering Django with Background Workers
class: text-center
highlighter: shiki
transition: slide-left
mdc: true
2024-05-10 11:25:05 +01:00
monaco: false
2024-04-19 11:31:54 +01:00
themeConfig:
primary: '#0c4b33'
2024-05-10 16:47:07 +01:00
addons:
- "slidev-addon-qrcode"
2024-04-19 11:31:54 +01:00
---
# Empowering Django with Background Workers
2024-05-20 17:55:31 +01:00
### Jake Howard{.mt-10}
2024-04-19 11:31:54 +01:00
2024-05-20 17:55:31 +01:00
<ul class="list-none! text-sm [&>li]:m-0!">
2024-04-19 11:31:54 +01:00
<li>Senior Systems Engineer @ Torchbox <mdi-fire class="fill-white"/></li>
2024-05-29 14:39:51 +01:00
<li>Core, Security & Performance teams @ Wagtail <logos-wagtail class="fill-white"/></li>
2024-04-19 11:31:54 +01:00
</ul>
2024-05-20 17:55:31 +01:00
<ul class="list-none! text-xs [&>li]:m-0! mt-5">
2024-04-19 11:31:54 +01:00
<li><mdi-earth /> theorangeone.net</li>
<li><mdi-twitter /> @RealOrangeOne</li>
<li><mdi-github /> @RealOrangeOne</li>
<li><mdi-mastodon /> @jake@theorangeone.net</li>
</ul>
2024-05-10 16:34:28 +01:00
2024-05-10 16:47:07 +01:00
<div class="absolute right-0 bottom-4">
<QRCode
:width="110"
:height="110"
data="https://0rng.one/dceu24"
:dotsOptions="{ color: 'white' }"
/>
</div>
2024-05-10 16:34:28 +01:00
---
layout: center
---
2024-05-24 12:55:29 +01:00
# Django is a web framework
2024-05-10 16:34:28 +01:00
```mermaid
flowchart LR
U(User 🧑‍💻)
D[\Django/]
U---->|Request|D
D---->|Response|U
```
<style>
.mermaid {
text-align: center;
}
</style>
---
layout: full
---
2024-05-24 12:55:29 +01:00
# Django isn't _just_ for websites
2024-05-10 16:34:28 +01:00
```mermaid
2024-05-24 12:55:29 +01:00
flowchart TD
2024-05-10 16:34:28 +01:00
U[User 🧑‍💻]
D[\Django/]
DB[(Database)]
E>Email]
EA[External API]
2024-05-24 12:55:29 +01:00
V[[Video Transcoding]]
R[Reporting]
ML((Machine<br>Learning))
2024-05-10 16:34:28 +01:00
2024-05-24 12:55:29 +01:00
U<--->D
2024-05-10 16:34:28 +01:00
2024-05-24 12:55:29 +01:00
D---DB
D-..-E & EA & V & R & ML
2024-05-10 16:34:28 +01:00
```
2024-05-24 12:55:29 +01:00
<style>
.mermaid {
text-align: center;
}
</style>
2024-05-10 16:34:28 +01:00
---
layout: full
---
2024-05-24 12:55:29 +01:00
# Background Workers!
2024-05-10 16:34:28 +01:00
```mermaid
2024-05-24 12:55:29 +01:00
flowchart TD
2024-05-10 16:34:28 +01:00
U[User 🧑‍💻]
D[\Django/]
2024-05-24 12:55:29 +01:00
2024-05-10 16:34:28 +01:00
E>Email]
EA[External API]
2024-05-24 12:55:29 +01:00
V[[Video Transcoding]]
R[Reporting]
ML((Machine<br>Learning))
2024-05-10 16:34:28 +01:00
2024-05-24 12:55:29 +01:00
B{{<strong>Background Processing</strong>}}
2024-05-10 16:34:28 +01:00
2024-05-24 12:55:29 +01:00
U<-->D
2024-05-10 16:34:28 +01:00
2024-05-24 12:55:29 +01:00
D-..-B
2024-05-10 16:34:28 +01:00
2024-05-24 12:55:29 +01:00
B---E & EA & V & R & ML
2024-05-10 16:34:28 +01:00
```
2024-05-24 12:55:29 +01:00
<style>
.mermaid {
text-align: center;
}
</style>
2024-05-10 16:34:28 +01:00
---
2024-05-24 12:55:29 +01:00
layout: section
2024-05-10 16:34:28 +01:00
---
# Background Workers?
---
layout: fact
---
```mermaid
flowchart LR
D[\Django/]
S[(Queue Store)]
R1{Runner}
R2{Runner}
R3{Runner}
2024-05-24 12:55:29 +01:00
D<----->S<-....->R1 & R2 & R3
2024-05-10 16:34:28 +01:00
```
---
2024-05-24 12:55:29 +01:00
layout: section
2024-05-10 16:34:28 +01:00
---
# When?
---
layout: fact
---
```mermaid
flowchart BT
D[\Django/]
subgraph Fast & Reliable
DB[(Database)]
C[(Cache)]
end
subgraph Slow / Unreliable
E>Email]
EA[External API]
V[[Video Transcode]]
ML((Machine Learning))
end
D---DB & C
D-.-E & EA & V & ML
```
---
2024-05-24 12:55:29 +01:00
layout: section
2024-05-10 16:34:28 +01:00
---
# Background Workers _in Django_
---
layout: cover
2024-05-24 12:55:29 +01:00
background: /celery.svg
2024-05-10 16:34:28 +01:00
---
# Celery!
<style>
.slidev-layout {
background: white;
2024-05-24 12:55:29 +01:00
background-size: contain !important;
2024-05-10 16:34:28 +01:00
}
</style>
---
layout: image-right
image: https://images.unsplash.com/photo-1444703686981-a3abbc4d4fe3?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D
---
# Others...
<Transform :scale="1.05">
- ~~Celery~~
- arq
- Django DB Queue
- Django Lightweight Queue
- Django Too Simple Q
- Django-Q
- Django-Q2
- Dramatiq
- Huey
- RQ
- Taskiq
- ...
</Transform>
---
layout: cover
background: https://images.unsplash.com/photo-1522096823084-2d1aa8411c13?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D
---
# Email <mdi-email-fast-outline />
---
2024-05-20 17:56:30 +01:00
layout: center
2024-05-10 16:34:28 +01:00
---
2024-05-24 12:55:29 +01:00
# Sending an email
```python {all|7|8|9-14|all}
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
from wagtail.models import Page
for user in page.subscribers.iterator():
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
send_mail(
subject=f"A change to {page.title} has been published",
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
```
---
layout: center
---
2024-05-17 11:02:02 +01:00
```python {all|18|19|10|11-16|all|18-19|all}
2024-05-10 16:34:28 +01:00
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
import django_rq
2024-05-17 11:02:02 +01:00
from wagtail.models import Page
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
2024-05-10 16:34:28 +01:00
send_mail(
2024-05-24 12:55:29 +01:00
subject=f"A change to {page.title} has been published",
2024-05-10 16:34:28 +01:00
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
2024-05-17 11:02:02 +01:00
for user in page.subscribers.iterator():
2024-05-10 16:34:28 +01:00
django_rq.enqueue(send_email_to_user, user)
```
---
layout: center
---
2024-05-24 12:55:29 +01:00
# Using <span v-click.hide="1">RQ</span><span v-click="1"><s class="opacity-60">RQ</s> Celery</span>
2024-05-10 16:34:28 +01:00
````md magic-move
```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
import django_rq
2024-05-17 11:02:02 +01:00
from wagtail.models import Page
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
2024-05-10 16:34:28 +01:00
send_mail(
2024-05-24 12:55:29 +01:00
subject=f"A change to {page.title} has been published",
2024-05-10 16:34:28 +01:00
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
2024-05-17 11:02:02 +01:00
for user in page.subscribers.iterator():
2024-05-10 16:34:28 +01:00
django_rq.enqueue(send_email_to_user, user)
```
2024-05-17 11:02:02 +01:00
```python {all|7-9,20|all}
2024-05-10 16:34:28 +01:00
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
2024-05-17 11:02:02 +01:00
from wagtail.models import Page
2024-05-10 16:34:28 +01:00
from my_celery_config import app
@app.task
2024-05-17 11:02:02 +01:00
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
2024-05-10 16:34:28 +01:00
send_mail(
2024-05-24 12:55:29 +01:00
subject=f"A change to {page.title} has been published",
2024-05-10 16:34:28 +01:00
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
2024-05-17 11:02:02 +01:00
for user in page.subscribers.iterator():
2024-05-10 16:34:28 +01:00
send_email_to_user.delay(user)
```
````
2024-05-24 12:55:29 +01:00
<style>
.slidev-vclick-hidden {
display: none;
}
</style>
2024-05-10 16:34:28 +01:00
---
2024-05-20 17:56:55 +01:00
layout: image
image: /situation.png
backgroundSize: 50%
2024-05-10 16:34:28 +01:00
---
---
layout: image
image: /ridiculous.png
2024-05-20 17:56:55 +01:00
backgroundSize: 49%
2024-05-10 16:34:28 +01:00
---
---
layout: fact
---
2024-05-17 11:02:02 +01:00
## Introducing*:{.mb-5}
2024-05-10 16:34:28 +01:00
2024-05-20 17:56:55 +01:00
# `django.tasks`
2024-05-10 16:47:07 +01:00
2024-05-24 12:55:29 +01:00
<div class="absolute right-1/2 translate-x-1/2 mt-12">
2024-05-10 16:47:07 +01:00
<QRCode
:width="120"
:height="120"
data="https://pypi.org/project/django-tasks/"
:dotsOptions="{ color: 'white' }"
/>
</div>
2024-05-17 11:02:02 +01:00
---
layout: image-right
image: https://images.unsplash.com/photo-1674027444485-cec3da58eef4?q=80&w=1932&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D
class: flex items-center text-xl
---
2024-05-10 16:34:28 +01:00
2024-05-17 11:02:02 +01:00
- API contract between library and application developers
- Swappable backends through `settings.py`
- Built in backends:
- ORM
- "Immediate"
- "Dummy"
- Django 5.2 🤞
- Backport for 4.2+
2024-05-10 16:34:28 +01:00
---
layout: center
---
2024-05-24 12:55:29 +01:00
# <span v-click.hide="1">Using Celery</span><span v-click="1">Using <code>django.tasks</code></span>
2024-05-10 16:34:28 +01:00
````md magic-move
```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
2024-05-17 11:02:02 +01:00
from wagtail.models import Page
2024-05-10 16:34:28 +01:00
from my_celery_config import app
@app.task
2024-05-17 11:02:02 +01:00
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
2024-05-10 16:34:28 +01:00
send_mail(
2024-05-24 12:55:29 +01:00
subject=f"A change to {page.title} has been published",
2024-05-10 16:34:28 +01:00
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
2024-05-17 11:02:02 +01:00
for user in page.subscribers.iterator():
2024-05-10 16:34:28 +01:00
send_email_to_user.delay(user)
```
2024-05-17 11:02:02 +01:00
```python {all|7-9,20|all}
2024-05-10 16:34:28 +01:00
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
2024-05-17 11:02:02 +01:00
from wagtail.models import Page
2024-05-10 16:34:28 +01:00
from django.tasks import task
@task()
2024-05-17 11:02:02 +01:00
def send_email_to_user(page: Page, user: User):
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
2024-05-10 16:34:28 +01:00
send_mail(
2024-05-24 12:55:29 +01:00
subject=f"A change to {page.title} has been published",
2024-05-10 16:34:28 +01:00
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
2024-05-17 11:02:02 +01:00
for user in page.subscribers.iterator():
2024-05-10 16:34:28 +01:00
send_email_to_user.enqueue(user)
```
````
2024-05-24 12:55:29 +01:00
<style>
.slidev-vclick-hidden {
display: none;
}
</style>
2024-05-10 16:34:28 +01:00
---
layout: center
---
```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string
2024-05-24 12:55:29 +01:00
from wagtail.models import Page
2024-05-10 16:34:28 +01:00
2024-05-17 11:02:02 +01:00
for user in page.subscribers.iterator():
2024-05-24 12:55:29 +01:00
email_content = render_to_string("notification-email.html", {"user": user, "page": page})
2024-05-10 16:34:28 +01:00
send_mail(
2024-05-24 12:55:29 +01:00
subject=f"A change to {page.title} has been published",
2024-05-10 16:34:28 +01:00
message=email_content
from_email=None, # Use the default sender email
recipient_list=[user.email]
)
```
2024-05-20 17:57:44 +01:00
<br />
2024-05-10 16:34:28 +01:00
<v-click>
```python
# settings.py
EMAIL_BACKEND = "django.core.mail.backends.tasks.SMTPEmailBackend"
```
</v-click>
---
layout: image-right
image: /soon.png
class: flex justify-center text-2xl flex-col
---
# Q: Why something new?
---
2024-05-17 11:02:02 +01:00
layout: image-right
image: https://images.unsplash.com/photo-1525683879097-8babce1c602a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D
2024-05-24 12:55:29 +01:00
class: flex justify-center text-xl flex-col
2024-05-10 16:34:28 +01:00
---
2024-05-17 11:02:02 +01:00
# Q: Why something built-in?
2024-05-10 16:34:28 +01:00
2024-05-10 17:11:55 +01:00
- Reduce barrier to entry
2024-05-24 12:55:29 +01:00
- Reduce cognitive load
- Reduce complexity for smaller projects
- Improve interoperability
- Use what's already there
- A common API
2024-05-10 16:34:28 +01:00
2024-05-10 17:11:55 +01:00
---
layout: center
2024-05-17 11:02:02 +01:00
transition: fade
2024-05-10 17:11:55 +01:00
---
2024-05-24 12:55:29 +01:00
![](/celery.svg){.h-32.mx-auto}
2024-05-10 17:11:55 +01:00
## vs
2024-05-24 12:55:29 +01:00
![](/postgres.png){.h-36.mx-auto}
2024-05-17 11:02:02 +01:00
<style>
.slidev-layout {
background: white;
color: black;
text-align: center;
}
</style>
---
layout: center
---
2024-05-24 12:55:29 +01:00
![](/elasticsearch.png){.h-32.mx-auto}
2024-05-10 17:11:55 +01:00
2024-05-17 11:02:02 +01:00
## vs
2024-05-24 12:55:29 +01:00
![](/postgres.png){.max-h-36.mx-auto}
2024-05-17 11:02:02 +01:00
2024-05-10 17:11:55 +01:00
<style>
.slidev-layout {
background: white;
color: black;
text-align: center;
}
</style>
2024-05-10 16:34:28 +01:00
---
layout: section
---
# Where are we now?
<v-click>
## `pip install django-tasks`
</v-click>
2024-05-17 11:02:02 +01:00
---
layout: section
---
# Where will we be soon™?
2024-05-10 16:34:28 +01:00
---
2024-05-10 17:26:20 +01:00
layout: cover
2024-05-24 12:55:29 +01:00
background: /celery.svg
2024-05-10 16:34:28 +01:00
---
2024-05-10 17:26:20 +01:00
# Is this the end?
<style>
.slidev-layout {
background: white;
2024-05-24 12:55:29 +01:00
background-size: contain !important;
2024-05-10 17:26:20 +01:00
}
</style>
2024-05-10 16:34:28 +01:00
---
layout: image-right
image: https://images.unsplash.com/photo-1451187580459-43490279c0fa?q=80&w=1744&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D
class: flex justify-center flex-col text-xl
---
# Out of scope
- Completion / failed hooks
- Bulk queueing
- Automated task retrying
- Task runner API
- Unified observability
- Cron-based scheduling
- Task timeouts
- Swappable argument serialization
2024-05-24 12:55:29 +01:00
- ...
2024-05-10 16:34:28 +01:00
2024-05-17 11:02:02 +01:00
---
layout: cover
background: https://images.unsplash.com/photo-1519187903022-c0055ec4036a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D
---
# The future is bright
2024-05-10 16:34:28 +01:00
---
layout: section
---
# What's next?
2024-05-24 12:55:29 +01:00
<v-click>
## `pip install django-tasks`
<div class="absolute right-1/2 translate-x-1/2 mt-12">
<QRCode
:width="120"
:height="120"
data="https://pypi.org/project/django-tasks/"
:dotsOptions="{ color: 'white' }"
/>
</div>
</v-click>
2024-05-10 16:34:28 +01:00
---
layout: end
---
2024-05-10 16:47:07 +01:00
2024-05-24 12:55:29 +01:00
END
2024-05-10 16:47:07 +01:00
<div class="absolute right-1/2 bottom-4 translate-x-1/2">
<QRCode
:width="110"
:height="110"
data="https://0rng.one/dceu24"
:dotsOptions="{ color: 'white' }"
/>
</div>