--- title: Bringing Background Workers to Django class: text-center highlighter: shiki transition: slide-left mdc: true monaco: false themeConfig: primary: '#0c4b33' --- # Bringing Background Workers to ## Jake Howard{.mt-5} ### Django London{.mt-5} #### September 2024 --- layout: full --- # `$ whoami`
--- layout: center --- # Django is a web framework ```mermaid flowchart LR U(User 🧑‍💻) D[\Django/] U---->|Request|D D---->|Response|U ``` --- layout: full --- # Django isn't _just_ for websites ```mermaid flowchart BT U[User 🧑‍💻] D[\Django/] DB[(Database)] E>Email] EA[External API] V[[Video Transcoding]] R[Reporting] ML((Machine
Learning)) U<--->D D---DB D-..-E & EA & V & R & ML ``` --- layout: full --- # Background Workers? ```mermaid flowchart BT U[User 🧑‍💻] D[\Django/] E>Email] EA[External API] V[[Video Transcoding]] R[Reporting] ML((Machine
Learning)) B{{Background Worker}} U<-->D D-..-B B---E & EA & V & R & ML ``` --- layout: section --- ## Background worker architecture ```mermaid flowchart LR D[\Django/] S[(Queue Store)] R1{Runner} R2{Runner} R3{Runner} D<----->S<-....->R1 & R2 & R3 ``` --- layout: section --- # When? --- layout: cover background: https://images.unsplash.com/photo-1518729371765-043e54eb5674?q=80&w=1807&auto=format&fit=crop&ixlib=rb-4.0.3 --- # Does it take time?{.text-right} --- layout: fact --- ## Does it leave your infrastructure?{.mb-5} ```mermaid flowchart BT D[\Django/] subgraph Slow / Unreliable E>Email] EA[External API] V[[Video Transcode]] R[Reporting] ML((Machine
Learning)) end subgraph Fast & Reliable DB[(Database)] C[(Cache)] end D---DB & C D-.-E & EA & V & R & ML ``` --- layout: cover background: https://images.unsplash.com/photo-1518770660439-4636190af475?q=80&w=3870&auto=format&fit=crop&ixlib=rb-4.0.3 --- # Specialized hardware? --- layout: image-right image: https://images.unsplash.com/photo-1711606815631-38d32cdaec3e?q=80&w=2070&auto=format&fit=crop&ixlib=rb-4.0.3 class: text-xl --- # Examples - Compiling code - Complex reporting - File uploads - Model training - PDF generation - Resizing images - Sending email - Transcoding video - ... 🤯 --- layout: section --- # Background Workers in --- layout: image-right image: https://images.unsplash.com/photo-1444703686981-a3abbc4d4fe3?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3 --- # Libraries - Celery

- arq - Django DB Queue - Django Lightweight Queue - Django Too Simple Q - Django-Q - Django-Q2 - Dramatiq - Huey - RQ - Taskiq - ... --- layout: cover background: https://images.unsplash.com/photo-1522096823084-2d1aa8411c13?q=80&w=1740&auto=format&fit=crop&ixlib=rb-4.0.3 --- ## Example: # Email --- layout: center --- # Sending an email ```python {all|7|8|9-14|all} from django.contrib.auth.models import User from django.core.mail import send_mail from django.template.loader import render_to_string from wagtail.models import Page for user in page.subscribers.iterator(): email_content = render_to_string("notification-email.html", {"user": user, "page": page}) send_mail( subject=f"A change to {page.title} has been published", message=email_content from_email=None, # Use the default sender email recipient_list=[user.email] ) ``` --- layout: center --- ```python {all|18|19|10|11-16|all|18-19|all} from django.contrib.auth.models import User from django.core.mail import send_mail from django.template.loader import render_to_string import django_rq from wagtail.models import Page def send_email_to_user(page: Page, user: User): email_content = render_to_string("notification-email.html", {"user": user, "page": page}) send_mail( subject=f"A change to {page.title} has been published", message=email_content from_email=None, # Use the default sender email recipient_list=[user.email] ) for user in page.subscribers.iterator(): django_rq.enqueue(send_email_to_user, user) ``` --- layout: center --- # Using RQRQ Celery ````md magic-move ```python from django.contrib.auth.models import User from django.core.mail import send_mail from django.template.loader import render_to_string import django_rq from wagtail.models import Page def send_email_to_user(page: Page, user: User): email_content = render_to_string("notification-email.html", {"user": user, "page": page}) send_mail( subject=f"A change to {page.title} has been published", message=email_content from_email=None, # Use the default sender email recipient_list=[user.email] ) for user in page.subscribers.iterator(): django_rq.enqueue(send_email_to_user, user) ``` ```python {all|7-9,20|all} from django.contrib.auth.models import User from django.core.mail import send_mail from django.template.loader import render_to_string from wagtail.models import Page from my_celery_config import app @app.task def send_email_to_user(page: Page, user: User): email_content = render_to_string("notification-email.html", {"user": user, "page": page}) send_mail( subject=f"A change to {page.title} has been published", message=email_content from_email=None, # Use the default sender email recipient_list=[user.email] ) for user in page.subscribers.iterator(): send_email_to_user.delay(user) ``` ```` --- layout: image image: /situation.png backgroundSize: 50% --- --- layout: image image: /ridiculous.png backgroundSize: 49% --- --- layout: fact --- ## Introducing*:{.mb-5 .mt-3} # `django.tasks`
--- layout: image-right image: https://images.unsplash.com/photo-1674027444485-cec3da58eef4?q=80&w=1932&auto=format&fit=crop&ixlib=rb-4.0.3 class: flex items-center text-xl --- - API contract between library and application developers - Swappable backends through `settings.py` - Built in implementations: - ORM - "Immediate" - "Dummy" - Django 5.2 🤞 - Backport for 4.2+ --- layout: center --- # Using CeleryUsing django.tasks ````md magic-move ```python from django.contrib.auth.models import User from django.core.mail import send_mail from django.template.loader import render_to_string from wagtail.models import Page from my_celery_config import app @app.task def send_email_to_user(page: Page, user: User): email_content = render_to_string("notification-email.html", {"user": user, "page": page}) send_mail( subject=f"A change to {page.title} has been published", message=email_content from_email=None, # Use the default sender email recipient_list=[user.email] ) for user in page.subscribers.iterator(): send_email_to_user.delay(user) ``` ```python from django.contrib.auth.models import User from django.core.mail import send_mail from django.template.loader import render_to_string from wagtail.models import Page from django.tasks import task @task() def send_email_to_user(page_id: int, user_id: int): page = Page.objects.get(id=page_id) user = User.objects.get(id=user_id) email_content = render_to_string("notification-email.html", {"user": user, "page": page}) send_mail( subject=f"A change to {page.title} has been published", message=email_content from_email=None, # Use the default sender email recipient_list=[user.email] ) for user in page.subscribers.iterator(): send_email_to_user.enqueue(user.id) ``` ```` --- layout: image-right image: /soon.png class: flex justify-center text-2xl flex-col --- # Q: Why something new? --- layout: image-right image: https://images.unsplash.com/photo-1525683879097-8babce1c602a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3 class: flex justify-center text-xl flex-col --- # Q: Why something built-in? - Reduce barrier to entry - Reduce cognitive load - Reduce complexity for smaller projects - Improve interoperability - Use what's already there - A common API --- layout: center transition: fade --- ![](/celery.svg){.h-32.mx-auto} ## vs ![](/postgres.png){.h-36.mx-auto} --- layout: center --- ![](/elasticsearch.png){.h-32.mx-auto} ## vs ![](/postgres.png){.max-h-36.mx-auto} --- layout: section --- # Where are we now? --- layout: image image: /dep.png --- --- layout: image image: /pypi.png --- --- layout: section --- # Where will we be _soon_™? --- layout: cover background: /celery.svg --- # Is this the end? --- layout: image-right image: https://images.unsplash.com/photo-1451187580459-43490279c0fa?q=80&w=1744&auto=format&fit=crop&ixlib=rb-4.0.3 class: flex justify-center flex-col text-xl --- # Out of scope (_for now_) - Completion / failed hooks - Bulk queueing - Automated task retrying - Task runner API - Unified observability - Cron-based scheduling - Task timeouts - Swappable argument serialization - ... --- layout: cover background: https://images.unsplash.com/photo-1519187903022-c0055ec4036a?q=80&w=1335&auto=format&fit=crop&ixlib=rb-4.0.3 --- # The future is bright --- layout: section --- # What's next? ## `pip install django-tasks`
--- layout: section class: text-center text-2xl --- # Let's chat!
  • theorangeone.net
  • @RealOrangeOne
  • @RealOrangeOne
  • @jake@theorangeone.net

Me

django-tasks

--- layout: end --- END