slides/bringing-background-workers-to-django
1
Fork 0
forked from slides/empowering-django-with-background-workers
Code Activity Actions

Tweak script, yet again

Browse source
This commit is contained in:
Jake Howard 2024-06-05 09:48:04 +01:00
parent cd626caaba
commit d751b46308
Signed by: jake
GPG key ID: 57AFB45680EDD477
1 changed files with 39 additions and 24 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

63
slides.md
View file

@ -66,7 +66,7 @@ flowchart LR
- Django is a web framework
- It's a magic box which turns HTTP requests into HTTP responses
- What you do inside that box is up to you
- For something like a blog, that's probably as far as it needs to go
- For something like a blog, that's probably as far as it needs to go
-->
---
@ -150,11 +150,11 @@ flowchart TD
<!--
- You need a background worker
- But[click]
- What are background workers
- What _are_ background workers
- Let you offload complexity outside of the request-response cycle
- To be run somewhere else, potentially at a later date
- They keep requests quick
- They keep requests nice and fast
- Move the slow bits somewhere else
- User doesn't have to wait
- Improves throughput and latency
@ -195,9 +195,10 @@ layout: section
- Background workers are very useful tool
- But that doesn't mean they're useful for everything, all the time
- As with all great things: "It depends"
- As with all great things: "It depends" when they're useful
- Trade-off between complexity and functionality
- A few things to consider
- If you're considering whether an action makes sense in the background:
- There are a few things to consider...
-->
---
@ -209,7 +210,7 @@ background: https://images.unsplash.com/photo-1518729371765-043e54eb5674?q=80&w=
<!--
- Does it take time
- _Could_ it take time
- Or _could_ it take time
- Don't want to make the user wait
- Unable to close the tab or do something else
- Go off and do it in the background, and let them know whether it's done
@ -245,7 +246,7 @@ flowchart BT
```
<!--
- Leaving your infrastructure
- Does control leave your infrastructure?
- The core components (Server, DB, Cache etc) you control and can closely monitor
- And are in a good position to fix it if something goes wrong
- That's not true for external APIs
@ -312,8 +313,8 @@ layout: section
<!--
- Back to Django
- This is djangocon after all
- In Python and Django, there are lots of different frameworks to achieve this
- This is Djangocon after all
- In Python and Django, there are lots of different frameworks to achieve background workers
-->
---
@ -338,7 +339,7 @@ image: https://images.unsplash.com/photo-1444703686981-a3abbc4d4fe3?q=80&w=1740&
<!--
- All require an external library
- And possibly some external infrastructure
- And possibly some external infrastructure
- Celery is probably the biggest one
- But it's not all that exists
- So many different libraries exist
@ -550,6 +551,7 @@ backgroundSize: 50%
- Do you write and maintain integrations for _all_ task libraries
- Do you choose the big one(s) and force your users' hands?
- Do you expose a hook and let your users integrate themselves?
- It adds a huge maintenance burden, whichever you choose
- There isn't really a right answer
-->
@ -565,7 +567,8 @@ backgroundSize: 49%
- A single API to help developers use a library
- Without tieing their hands
- First-party
- Allowing library developers to depend on it instead of supporting every separate API
- Allowing library developers to depend on it
- Instead of supporting every separate API
- Scale easily as your needs change
- Be easy to get started with for small projects
- But feature-packed for larger deployments
@ -597,7 +600,7 @@ class: flex items-center text-xl
- API contract between library and application developers
- Swappable backends through `settings.py`
- Built in backends:
- Built in implementations:
- ORM
- "Immediate"
- "Dummy"
@ -608,8 +611,8 @@ class: flex items-center text-xl
- An API contract between worker library maintainers and application developers
- Compatibility layer between Django and their native APIs
- Hopefully the promise of "Write once, run anywhere"
- Built-in worker queues
- ORM based (production grade)
- Built-in implementations
- ORM based (production grade, building on the power of the ORM)
- "Immediate" (ie doesn't background anything) loaded by default
- Dummy (for testing)
- Hopefully landing in Django 5.2
@ -680,13 +683,14 @@ for user in page.subscribers.iterator():
- Let's look at the same code example as before
- This is tied to Celery
- If want to support RQ too, I'd have to duplicate some parts
- Instead, let's rewrite this once to use `django.tasks`[click]
- Instead, let's write this once to use `django.tasks`[click]
- Still simple, clear, approachable and easy to use
- If I say so myself
- If we swapped to RQ: 0 lines need to change
- If a new library comes out, 0 lines need to change
- If this is in a library, not my own code, I'm not constrained by their preferences
- And the maintainer doesn't have extra work to support my preferences
- They can use what they like, I can use what I like
- For testing, I can use an in-memory backend
- With 0 lines changed
-->
@ -726,9 +730,9 @@ EMAIL_BACKEND = "django.core.mail.backends.tasks.SMTPEmailBackend"
<!--
- In this case, we can actually make it even easier
- Because email is such a common use case, and so easy to extract
- Go back to the simple implementation
- Let's go back to the simple implementation
- No background workers in sight
- [click]Use the built-in task email backend
- [click]Instead, we can change the email backend
- Emails are magically sent in the background automatically
- Without additional work
-->
@ -753,6 +757,7 @@ class: flex justify-center text-2xl flex-col
- The built-in version will hopefully become great
- But must be done with careful planning and consideration
- Django needs to remain the stable and reliable base it always has been
- Don't want to burn-out the Django maintainers
-->
---
@ -778,9 +783,10 @@ class: flex justify-center text-xl flex-col
- Scale with your needs
- A developer can join a new project and already be productive
- A common API also helps library maintainers
- Maintaining a large library is work enough
- Maintaining a library is work enough
- Without needing to think about how to move code to the background
- If Django can take complexity off you, great
- Use the tools provided to you
- Currently, it's not really an option
- The burden is too great
- No additional dependencies for your library
@ -890,6 +896,11 @@ layout: section
- Tell me about all the bugs in my code
- The more testing we can do now, the better
- There's still work to do
- Features
- Improvements
- Performance
- Scalability
- etc etc
-->
---
@ -943,7 +954,7 @@ image: https://images.unsplash.com/photo-1451187580459-43490279c0fa?q=80&w=1744&
class: flex justify-center flex-col text-xl
---
# Out of scope
# Out of scope (_for now_)
- Completion / failed hooks
- Bulk queueing
@ -964,6 +975,7 @@ class: flex justify-center flex-col text-xl
- But I hope we can slowly catch them up
- Bringing the stability and longevity guarantees that come with Django
- Doesn't mean they'll never come
- With your help, we can make these happen
-->
---
@ -981,11 +993,12 @@ background: https://images.unsplash.com/photo-1519187903022-c0055ec4036a?q=80&w=
- Improve throughput
- Reduce latency
- Improve reliability
- Gone are the days of needing additional research and testing to find the tooling you need
- You can use the ones built-in to Django
- And as you scale, it's easy to change
- _without_ rewriting half your application
- With all the knowledge to make an informed decision
- Gone are the days of needing additional research and testing to find the tooling you need
- You can use the ones built-in to Django
- And as you scale, it's easy to change
- _without_ rewriting half your application
- With all the knowledge to make an informed decision
- A lower barrier helps everyone!
-->
---
@ -1005,10 +1018,12 @@ layout: section
- Test it out
- Report back your issues
- Suggest improvements
- Issues / Discussions are open
- If you want to get involved, please do!
- There's plenty of work to do
- And I can't do it alone!
- If you maintain a worker library
- Have interesting use cases
- Or have been burned by one...
-->