slides/bringing-background-workers-to-django
1
Fork 0
forked from slides/empowering-django-with-background-workers
Code Activity Actions
bringing-background-workers.../slides.md
Jake Howard 07d4b6f274
All checks were successful
/ build (push) Successful in 1m9s
Details
Final script tweaks
2024-09-10 22:27:09 +01:00

28 KiB
Raw Blame History

title class highlighter transition mdc monaco themeConfig
Bringing Background Workers to Django text-center shiki slide-left true false
primary
#0c4b33

Bringing Background Workers to

Jake Howard

Django London

September 2024

layout: full

$ whoami

  • Senior Systems Engineer @ Torchbox
  • Core, Security & Performance teams @ Wagtail
  • Django Software Foundation Member
  • I'm an avid self-hoster
  • I help students build robots in my "spare" time (https://srobo.org)
  • theorangeone.net
  • @RealOrangeOne
  • @jake@theorangeone.net
  • @RealOrangeOne

layout: center

Django is a web framework

flowchart LR
    U(User 🧑‍💻)
    D[\Django/]

    U---->|Request|D
    D---->|Response|U

layout: full

Django isn't just for websites

flowchart BT
    U[User 🧑‍💻]
    D[\Django/]
    DB[(Database)]
    E>Email]
    EA[External API]
    V[[Video Transcoding]]
    R[Reporting]
    ML((Machine<br>Learning))

    U<--->D

    D---DB
    D-..-E & EA & V & R & ML

layout: full

Background Workers?

flowchart BT
    U[User 🧑‍💻]
    D[\Django/]

    E>Email]
    EA[External API]
    V[[Video Transcoding]]
    R[Reporting]
    ML((Machine<br>Learning))

    B{{<strong>Background Worker</strong>}}

    U<-->D

    D-..-B

    B---E & EA & V & R & ML

layout: section

Background worker architecture

flowchart LR
    D[\Django/]
    S[(Queue Store)]
    R1{Runner}
    R2{Runner}
    R3{Runner}

    D<----->S<-....->R1 & R2 & R3

layout: section

When?

layout: cover background: https://images.unsplash.com/photo-1518729371765-043e54eb5674?q=80&w=1807&auto=format&fit=crop&ixlib=rb-4.0.3

Does it take time?

layout: fact

Does it leave your infrastructure?

flowchart BT
    D[\Django/]

    subgraph Slow / Unreliable
      E>Email]
      EA[External API]
      V[[Video Transcode]]
      R[Reporting]
      ML((Machine<br>Learning))
    end

    subgraph Fast & Reliable
      DB[(Database)]
      C[(Cache)]
    end

    D---DB & C

    D-.-E & EA & V & R & ML

layout: cover background: https://images.unsplash.com/photo-1518770660439-4636190af475?q=80&w=3870&auto=format&fit=crop&ixlib=rb-4.0.3

Does it require specialized hardware?

layout: image-right image: https://images.unsplash.com/photo-1711606815631-38d32cdaec3e?q=80&w=2070&auto=format&fit=crop&ixlib=rb-4.0.3 class: text-xl

Examples

  • Compiling code
  • Complex reporting
  • File uploads
  • Model training
  • PDF generation
  • Resizing images
  • Sending email
  • Transcoding video
  • ... 🤯

layout: section

Background Workers in

layout: image-right image: https://images.unsplash.com/photo-1444703686981-a3abbc4d4fe3?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3

Libraries

  • Celery

  • arq
  • Django DB Queue
  • Django Lightweight Queue
  • Django Too Simple Q
  • Django-Q
  • Django-Q2
  • Dramatiq
  • Huey
  • RQ
  • Taskiq
  • ...

layout: cover background: https://images.unsplash.com/photo-1522096823084-2d1aa8411c13?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3

Example:

Email

layout: center

Sending an email

from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from wagtail.models import Page

for user in page.subscribers.iterator():
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

layout: center

from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

import django_rq

from wagtail.models import Page

def send_email_to_user(page: Page, user: User):
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    django_rq.enqueue(send_email_to_user, user)

layout: center

Using RQRQ Celery

```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

import django_rq

from wagtail.models import Page

def send_email_to_user(page: Page, user: User):
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    django_rq.enqueue(send_email_to_user, user)
```

```python {all|7-9,20|all}
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from wagtail.models import Page

from my_celery_config import app

@app.task
def send_email_to_user(page: Page, user: User):
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    send_email_to_user.delay(user)
```

layout: image image: /situation.png backgroundSize: 50%

layout: image image: /ridiculous.png backgroundSize: 49%

layout: fact

Introducing*:

django.tasks

layout: image-right image: https://images.unsplash.com/photo-1674027444485-cec3da58eef4?q=80&w=1932&auto=format&fit=crop&ixlib=rb-4.0.3 class: flex items-center text-xl

  • API contract between library and application developers
  • Swappable backends through settings.py
  • Built in implementations:
    • ORM
    • "Immediate"
    • "Dummy"
  • Django 5.2 🤞
    • Backport for 4.2+

layout: center

Using CeleryUsing django.tasks

```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.template.loader import render_to_string

from wagtail.models import Page

from my_celery_config import app

@app.task
def send_email_to_user(page: Page, user: User):
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    send_email_to_user.delay(user)
```

```python
from django.contrib.auth.models import User
from django.core.mail import send_mail
from django.tasks import task
from django.template.loader import render_to_string

from wagtail.models import Page


@task()
def send_email_to_user(page_id: int, user_id: int):
    page = Page.objects.get(id=page_id)
    user = User.objects.get(id=user_id)
    email_content = render_to_string("notification-email.html", {"user": user, "page": page})
    send_mail(
      subject=f"A change to {page.title} has been published",
      message=email_content
      from_email=None,  # Use the default sender email
      recipient_list=[user.email]
    )

for user in page.subscribers.iterator():
    send_email_to_user.enqueue(user.id)
```

layout: image-right image: /soon.png class: flex justify-center text-2xl flex-col

Q: Why something new?

layout: image-right image: https://images.unsplash.com/photo-1525683879097-8babce1c602a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3 class: flex justify-center text-xl flex-col

Q: Why something built-in?

  • Reduce barrier to entry
  • Reduce cognitive load
  • Reduce complexity for smaller projects
  • Improve interoperability
  • Use what's already there
  • A common API

layout: center transition: fade

{.h-32.mx-auto}

vs

{.h-36.mx-auto}

layout: center

{.h-32.mx-auto}

vs

{.max-h-36.mx-auto}

layout: section

Where are we now?

layout: image image: /dep.png

layout: image image: /pypi.png

layout: center

Since Djangocon Europe 2024:


  • Database worker 🚀
  • Improved testing on SQLite, PostgreSQL and MySQL
  • Improved testing on Windows, macOS and Linux
  • Add manage.py check support
  • Django Admin support for database backend
  • Store enqueued, started, and finished at times
  • Store task exceptions (with stacktrace)
  • Define task priority range and semantics
  • Prune completed database tasks
  • Transaction management
  • Soft and hard worker termination

layout: section

Where will we be soon™?

layout: cover background: /celery.svg

Is this the end?

layout: image-right image: https://images.unsplash.com/photo-1451187580459-43490279c0fa?q=80&w=1744&auto=format&fit=crop&ixlib=rb-4.0.3 class: flex justify-center flex-col text-xl

Out of scope (for now)

  • Task dependencies
  • Bulk queueing
  • Automated task retrying
  • Task runner API
  • Unified observability
  • Cron-based scheduling
  • Task timeouts
  • Swappable argument serialization
  • ...

layout: cover background: https://images.unsplash.com/photo-1519187903022-c0055ec4036a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3

The future is bright

layout: section

What's next?

pip install django-tasks

layout: section class: text-center text-2xl

Let's chat!

  • theorangeone.net
  • @RealOrangeOne
  • @jake@theorangeone.net
  • @RealOrangeOne

Me

django-tasks

layout: end

END