empowering-django-with-background-workers/slides.md

---
title: Empowering Django with Background Workers
class: text-center
highlighter: shiki
transition: slide-left
mdc: true
monaco: false
themeConfig:
  primary: '#0c4b33'
---

# Empowering Django with Background Workers

### Jake Howard{.mt-10}

<ul class="list-none! text-sm [&>li]:m-0!">
  <li>Senior Systems Engineer @ Torchbox <mdi-fire class="fill-white"/></li>
  <li>Core, Security & Performance teams @ Wagtail <logos-wagtail class="fill-white"/></li>
</ul>

<ul class="list-none! text-xs [&>li]:m-0! mt-5">
  <li><mdi-earth /> theorangeone.net</li>
  <li><mdi-twitter /> @RealOrangeOne</li>
  <li><mdi-github /> @RealOrangeOne</li>
  <li><mdi-mastodon /> @jake@theorangeone.net</li>
</ul>

<div class="absolute right-0 bottom-4">
  <img src="/dceu24-qrcode.png" width="110px" />
</div>

---
layout: center
---

# Django is a web framework

```mermaid
flowchart LR
    U(User 🧑‍💻)
    D[\Django/]

    U---->|Request|D
    D---->|Response|U
```

<style>
  .mermaid {
    text-align: center;
  }
</style>

---
layout: full
---

# Django isn't _just_ for websites

```mermaid
flowchart TD
    U[User 🧑‍💻]
    D[\Django/]
    DB[(Database)]
    E>Email]
    EA[External API]
    V[[Video Transcoding]]
    R[Reporting]
    ML((Machine<br>Learning))

    U<--->D

    D---DB
    D-..-E & EA & V & R & ML
```

<style>
  .mermaid {
    text-align: center;
  }
</style>


---
layout: full
---

# Background Workers!

```mermaid
flowchart TD
    U[User 🧑‍💻]
    D[\Django/]

    E>Email]
    EA[External API]
    V[[Video Transcoding]]
    R[Reporting]
    ML((Machine<br>Learning))

    B{{<strong>Background Processing</strong>}}

    U<-->D

    D-..-B

    B---E & EA & V & R & ML
```

<style>
  .mermaid {
    text-align: center;
  }
</style>

---
layout: section
---

# Background Workers?

---
layout: fact
---

```mermaid
flowchart LR
    D[\Django/]
    S[(Queue Store)]
    R1{Runner}
    R2{Runner}
    R3{Runner}

    D<----->S<-....->R1 & R2 & R3
```

---
layout: section
---

# When?

---
layout: cover
background: https://images.unsplash.com/photo-1518729371765-043e54eb5674?q=80&w=1807&auto=format&fit=crop&ixlib=rb-4.0.3
---

# Does it take time?{.text-right}

---
layout: fact
---

## Does it leave your infrastructure?{.mb-5}

```mermaid
flowchart BT
    D[\Django/]

    subgraph Slow / Unreliable
      E>Email]
      EA[External API]
      V[[Video Transcode]]
      R[Reporting]
      ML((Machine<br>Learning))
    end

    subgraph Fast & Reliable
      DB[(Database)]
      C[(Cache)]
    end

    D---DB & C

    D-.-E & EA & V & R & ML
```

---
layout: cover
background: https://images.unsplash.com/photo-1518770660439-4636190af475?q=80&w=3870&auto=format&fit=crop&ixlib=rb-4.0.3
---

# Specialized hardware?

---
layout: cover
background: https://images.unsplash.com/photo-1711606815631-38d32cdaec3e?q=80&w=2070&auto=format&fit=crop&ixlib=rb-4.0.3
---

## Example:
# Complex reporting

---
layout: section
---

# Background Workers in

<logos-django class="[&>path]:fill-white! h-fit w-60 -mt-20"/>

---
layout: cover
background: /celery.svg
---

# Celery!

<style>
.slidev-layout {
    background: white;
    background-size: contain !important;
}
</style>

---
layout: image-right
image: https://images.unsplash.com/photo-1444703686981-a3abbc4d4fe3?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3
---

# Others...

<Transform :scale="1.05">

- ~~Celery~~
- arq
- Django DB Queue
- Django Lightweight Queue
- Django Too Simple Q
- Django-Q
- Django-Q2
- Dramatiq
- Huey
- RQ
- Taskiq
- ...

</Transform>

---
layout: cover
background: https://images.unsplash.com/photo-1522096823084-2d1aa8411c13?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3
---

## Example:
# Email <mdi-email-fast-outline />

---
layout: center
---

# Sending an email

```python {all|7|8|9-14|all}
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from wagtail.models import Page

for user in page.subscribers.iterator():
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )
```

---
layout: center
---

```python {all|18|19|10|11-16|all|18-19|all}
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

import django_rq

from wagtail.models import Page

def send_email_to_user(page: Page, user: User):
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    django_rq.enqueue(send_email_to_user, user)
```

---
layout: center
---

# Using <span v-click.hide="1">RQ</span><span v-click="1"><s class="opacity-60">RQ</s> Celery</span>

````md magic-move
```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

import django_rq

from wagtail.models import Page

def send_email_to_user(page: Page, user: User):
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    django_rq.enqueue(send_email_to_user, user)
```

```python {all|7-9,20|all}
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from wagtail.models import Page

from my_celery_config import app

@app.task
def send_email_to_user(page: Page, user: User):
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    send_email_to_user.delay(user)
```
````

<style>
  .slidev-vclick-hidden {
    display: none;
  }
</style>

---
layout: image
image: /situation.png
backgroundSize: 50%
---

---
layout: image
image: /ridiculous.png
backgroundSize: 49%
---

---
layout: fact
---

## Introducing*:{.mb-5}

# `django.tasks`

<div class="absolute right-1/2 translate-x-1/2 mt-6">
  <img src="/django-tasks-qrcode.png" width="110px" />
</div>

---
layout: image-right
image: https://images.unsplash.com/photo-1674027444485-cec3da58eef4?q=80&w=1932&auto=format&fit=crop&ixlib=rb-4.0.3
class: flex items-center text-xl
---

- API contract between library and application developers
- Swappable backends through `settings.py`
- Built in backends:
  - ORM
  - "Immediate"
  - "Dummy"
- Django 5.2 🤞
  - Backport for 4.2+
---
layout: center
---

# <span v-click.hide="1">Using Celery</span><span v-click="1">Using <code>django.tasks</code></span>

````md magic-move
```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from wagtail.models import Page

from my_celery_config import app

@app.task
def send_email_to_user(page: Page, user: User):
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    send_email_to_user.delay(user)
```

```python {all|7-9,20|all}
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from wagtail.models import Page

from django.tasks import task

@task()
def send_email_to_user(page: Page, user: User):
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    send_email_to_user.enqueue(user)
```
````

<style>
  .slidev-vclick-hidden {
    display: none;
  }
</style>

---
layout: center
---

```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from wagtail.models import Page

for user in page.subscribers.iterator():
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )
```

<br />

<v-click>

```python
# settings.py
EMAIL_BACKEND = "django.core.mail.backends.tasks.SMTPEmailBackend"
```

</v-click>

---
layout: image-right
image: /soon.png
class: flex justify-center text-2xl flex-col
---

# Q: Why something new?

---
layout: image-right
image: https://images.unsplash.com/photo-1525683879097-8babce1c602a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3
class: flex justify-center text-xl flex-col
---

# Q: Why something built-in?

- Reduce barrier to entry
- Reduce cognitive load
- Reduce complexity for smaller projects
- Improve interoperability
- Use what's already there
- A common API

---
layout: center
transition: fade
---

![](/celery.svg){.h-32.mx-auto}

## vs

![](/postgres.png){.h-36.mx-auto}


<style>
.slidev-layout {
    background: white;
    color: black;
    text-align: center;
}
</style>

---
layout: center
---

![](/elasticsearch.png){.h-32.mx-auto}

## vs

![](/postgres.png){.max-h-36.mx-auto}


<style>
.slidev-layout {
    background: white;
    color: black;
    text-align: center;
}
</style>

---
layout: section
---

# Where are we now?

---
layout: image
image: /dep.png
---

---
layout: cover
---

# `pip install django-tasks`

<div class="absolute right-1/2 translate-x-1/2 mt-12">
  <QRCode
    :width="120"
    :height="120"
    data="https://pypi.org/project/django-tasks/"
    :dotsOptions="{ color: 'white' }"
  />
</div>

---
layout: section
---

# Where will we be _soon_™?

---
layout: cover
background: /celery.svg
---

# Is this the end?

<style>
.slidev-layout {
    background: white;
    background-size: contain !important;
}
</style>


---
layout: image-right
image: https://images.unsplash.com/photo-1451187580459-43490279c0fa?q=80&w=1744&auto=format&fit=crop&ixlib=rb-4.0.3
class: flex justify-center flex-col text-xl
---

# Out of scope

- Completion / failed hooks
- Bulk queueing
- Automated task retrying
- Task runner API
- Unified observability
- Cron-based scheduling
- Task timeouts
- Swappable argument serialization
- ...

---
layout: cover
background: https://images.unsplash.com/photo-1519187903022-c0055ec4036a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3
---

# The future is bright

---
layout: section
---

# What's next?

<div class="absolute right-1/2 translate-x-1/2 mt-12">
  <QRCode
    :width="120"
    :height="120"
    data="https://pypi.org/project/django-tasks/"
    :dotsOptions="{ color: 'white' }"
  />
</div>

---
layout: end
---

END

<div class="absolute right-1/2 bottom-4 translate-x-1/2">
  <img src="/dceu24-qrcode.png" width="110px" />
</div>