forked from slides/empowering-django-with-background-workers
Refine speaker notes
This commit is contained in:
parent
1d1f846753
commit
a3d48c2be9
1 changed files with 109 additions and 88 deletions
145
slides.md
145
slides.md
|
@ -20,8 +20,8 @@ themeConfig:
|
||||||
|
|
||||||
<ul class="list-none! text-xs [&>li]:m-0! mt-5">
|
<ul class="list-none! text-xs [&>li]:m-0! mt-5">
|
||||||
<li><mdi-earth /> theorangeone.net</li>
|
<li><mdi-earth /> theorangeone.net</li>
|
||||||
<li><mdi-twitter /> @RealOrangeOne</li>
|
|
||||||
<li><mdi-github /> @RealOrangeOne</li>
|
<li><mdi-github /> @RealOrangeOne</li>
|
||||||
|
<li><mdi-twitter /> @RealOrangeOne</li>
|
||||||
<li><mdi-mastodon /> @jake@theorangeone.net</li>
|
<li><mdi-mastodon /> @jake@theorangeone.net</li>
|
||||||
</ul>
|
</ul>
|
||||||
|
|
||||||
|
@ -157,8 +157,8 @@ flowchart TD
|
||||||
|
|
||||||
- They keep requests quick
|
- They keep requests quick
|
||||||
- Move the slow bits somewhere else
|
- Move the slow bits somewhere else
|
||||||
- So the user doesn't have to wait
|
- User doesn't have to wait
|
||||||
- This improves throughput and latency
|
- Improves throughput and latency
|
||||||
-->
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
|
@ -197,7 +197,7 @@ layout: section
|
||||||
- But that doesn't mean they're useful for everything, all the time
|
- But that doesn't mean they're useful for everything, all the time
|
||||||
|
|
||||||
- As with all great things: "It depends"
|
- As with all great things: "It depends"
|
||||||
- The added complexity may not be worth it
|
- Trade-off between complexity and functionality
|
||||||
- A few things to consider
|
- A few things to consider
|
||||||
-->
|
-->
|
||||||
|
|
||||||
|
@ -209,11 +209,11 @@ background: https://images.unsplash.com/photo-1518729371765-043e54eb5674?q=80&w=
|
||||||
# Does it take time?{.text-right}
|
# Does it take time?{.text-right}
|
||||||
|
|
||||||
<!--
|
<!--
|
||||||
- Something which takes time
|
- Does it take time
|
||||||
- Something which _could_ take time
|
- _Could_ it take time
|
||||||
- Rather than have the user wait
|
- Don't want to make the user wait
|
||||||
- Unable to close the tab or do something else
|
- Unable to close the tab or do something else
|
||||||
- Go off and do it in the background, and let them know whether it's done
|
- Go off and do it in the background, and let them know whether it's done
|
||||||
- Even if that's by polling it in the browser
|
- Even if that's by polling it in the browser
|
||||||
-->
|
-->
|
||||||
|
|
||||||
|
@ -247,11 +247,11 @@ flowchart BT
|
||||||
|
|
||||||
<!--
|
<!--
|
||||||
- Leaving your infrastructure
|
- Leaving your infrastructure
|
||||||
- The core components (Server, DB, Cache etc) you control and can closely monitor
|
- The core components (Server, DB, Cache etc) you control and can closely monitor
|
||||||
- And are in a good position to fix it if something goes wrong
|
- And are in a good position to fix it if something goes wrong
|
||||||
- That's not true for external APIs
|
- That's not true for external APIs
|
||||||
- It's someone else's SRE team
|
- It's someone else's SRE team
|
||||||
- Their performance regressions shouldn't affect your app
|
- Their performance characteristics shouldn't affect your app
|
||||||
-->
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
|
@ -263,7 +263,7 @@ background: https://images.unsplash.com/photo-1518770660439-4636190af475?q=80&w=
|
||||||
|
|
||||||
<!--
|
<!--
|
||||||
- Maybe it's less about when, more about where?
|
- Maybe it's less about when, more about where?
|
||||||
- Maybe it's more about the hardware it runs on
|
- Maybe it's more about the hardware it runs on
|
||||||
- GPUs
|
- GPUs
|
||||||
- Loads of RAM
|
- Loads of RAM
|
||||||
- External hardware
|
- External hardware
|
||||||
|
@ -281,10 +281,9 @@ background: https://images.unsplash.com/photo-1711606815631-38d32cdaec3e?q=80&w=
|
||||||
<!--
|
<!--
|
||||||
- An example: Complex reporting
|
- An example: Complex reporting
|
||||||
- Something analytical, crunching lots of data
|
- Something analytical, crunching lots of data
|
||||||
- Initially, it might take a few seconds, which is fine
|
- It might be fast locally
|
||||||
- You build a CSV as part of the request with something like `pandas`
|
|
||||||
- As your application grows, there'll be more data, so it'll likely take a lot longer
|
- As your application grows, there'll be more data, so it'll likely take a lot longer
|
||||||
- Rather than force the user to wait, let them get the data when it's ready
|
- Rather than force the user to wait, let them get the data when it's ready
|
||||||
- They can get back on with their day
|
- They can get back on with their day
|
||||||
- Web servers can get back to processing other requests
|
- Web servers can get back to processing other requests
|
||||||
-->
|
-->
|
||||||
|
@ -381,15 +380,14 @@ for user in page.subscribers.iterator():
|
||||||
|
|
||||||
- [click]This works perfectly fine
|
- [click]This works perfectly fine
|
||||||
- Scales _relatively_ well
|
- Scales _relatively_ well
|
||||||
- But has some issues
|
- But has some issues
|
||||||
- If connecting to the email server takes a while, the user has to wait
|
- If connecting to the email server takes a while, the user has to wait
|
||||||
- Usually only a few ms
|
- Usually only a few ms
|
||||||
- Might take a few seconds
|
- Might take a few seconds
|
||||||
- Subsequent emails are delayed whilst we process the earlier ones
|
|
||||||
- If something goes wrong with one email, the others won't send
|
- If something goes wrong with one email, the others won't send
|
||||||
- What if your email gateway is down altogether - do your requests start erroring?
|
- What if your email gateway is down altogether - do your requests start erroring?
|
||||||
- How do you handle it if they do?
|
- How do you handle it if they do?
|
||||||
- That web worker (eg gunicorn) can't process any other requests until this is done
|
- That web worker (eg gunicorn) can't process any other requests until this is done
|
||||||
-->
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
|
@ -508,7 +506,7 @@ for user in page.subscribers.iterator():
|
||||||
- Its own features
|
- Its own features
|
||||||
- Its own configuration
|
- Its own configuration
|
||||||
- Its own caveats / implementation details
|
- Its own caveats / implementation details
|
||||||
- What if we wanted to switch to Celery?
|
- What if we wanted to use Celery instead?
|
||||||
- [click]Well, that's easy
|
- [click]Well, that's easy
|
||||||
- [click]Just change a few lines
|
- [click]Just change a few lines
|
||||||
- [click]But there in lies the problem
|
- [click]But there in lies the problem
|
||||||
|
@ -525,21 +523,20 @@ backgroundSize: 50%
|
||||||
<!--
|
<!--
|
||||||
- It's hard enough having multiple options
|
- It's hard enough having multiple options
|
||||||
- But how do you choose between them?
|
- But how do you choose between them?
|
||||||
- If you've used multiple tools, you probably know which is best for you
|
- Maybe you have experience with libraries already
|
||||||
- Do you have the time (and patience) to test each one out?
|
- Do you have the time (and patience) to test each one out?
|
||||||
- Maybe you already have a standard you need to work to
|
- Maybe you already have a standard you need to work to
|
||||||
- Do you need a background worker which supports `asyncio`?
|
- Maybe you need specific features
|
||||||
- If you're new to Django, do you really want to spend the time weighing them all up?
|
- If you're new to Django, do you really want to spend the time weighing them all up?
|
||||||
- Knowing it could bite you as you grow or need a specific feature
|
- Knowing it could bite you as you grow or need a specific feature
|
||||||
- Requiring a lot of time refactoring in future
|
- Requiring a lot of time refactoring in future
|
||||||
|
|
||||||
- What about library maintainers
|
- What about library maintainers
|
||||||
- What if you built a library which has some background task needs
|
|
||||||
- Like, say, Wagtail
|
- Like, say, Wagtail
|
||||||
- Do you write and maintain integrations for _all_ task libraries
|
- Do you write and maintain integrations for _all_ task libraries
|
||||||
- Do you choose the big one(s) and force your users' hands?
|
- Do you choose the big one(s) and force your users' hands?
|
||||||
- Do you expose a hook and let your users integrate themselves?
|
- Do you expose a hook and let your users integrate themselves?
|
||||||
- It adds a huge maintenance burden, whichever you choose
|
- It adds a huge maintenance burden, whichever you choose
|
||||||
- There isn't really a right answer
|
- There isn't really a right answer
|
||||||
-->
|
-->
|
||||||
|
|
||||||
|
@ -551,8 +548,10 @@ backgroundSize: 49%
|
||||||
|
|
||||||
<!--
|
<!--
|
||||||
- There _should_ be one universal standard which combines them all
|
- There _should_ be one universal standard which combines them all
|
||||||
- A single API to help developers use a library, without tieing their hands
|
- A single API to help developers use a library
|
||||||
- First-party, allowing library developers to depend on it instead of supporting every separate API
|
- Without tieing their hands
|
||||||
|
- First-party
|
||||||
|
- Allowing library developers to depend on it instead of supporting every separate API
|
||||||
- Scale easily as your needs change
|
- Scale easily as your needs change
|
||||||
- Be easy to get started with for small projects
|
- Be easy to get started with for small projects
|
||||||
- But feature-packed for larger deployments
|
- But feature-packed for larger deployments
|
||||||
|
@ -670,11 +669,11 @@ for user in page.subscribers.iterator():
|
||||||
- Instead, let's rewrite this once to use `django.tasks`[click]
|
- Instead, let's rewrite this once to use `django.tasks`[click]
|
||||||
- Still simple, clear, approachable and easy to use
|
- Still simple, clear, approachable and easy to use
|
||||||
- If I say so myself
|
- If I say so myself
|
||||||
- Now, in future, if we swapped to RQ (for whatever reason), exactly 0 lines need to change
|
- If we swapped to RQ: 0 lines need to change
|
||||||
- If a new library comes out, 0 lines need to change
|
- If a new library comes out, 0 lines need to change
|
||||||
- If this is in a library, not my own code, I can keep using the library no matter what worker I'm using
|
- If this is in a library, not my own code, I'm not constrained by their preferences
|
||||||
- And the maintainer doesn't need to special case
|
- And the maintainer doesn't have extra work to support my preferences
|
||||||
- If I want to test this code, I can swap out the backend to an in-memory one, and interrogate it
|
- For testing, I can use an in-memory backend
|
||||||
- With 0 lines changed
|
- With 0 lines changed
|
||||||
-->
|
-->
|
||||||
|
|
||||||
|
@ -735,7 +734,7 @@ class: flex justify-center text-2xl flex-col
|
||||||
- As I've been told whilst working on this DEP
|
- As I've been told whilst working on this DEP
|
||||||
- Why not just vendor something existing?
|
- Why not just vendor something existing?
|
||||||
- If not Celery, then something else
|
- If not Celery, then something else
|
||||||
- That's not really the goal
|
- That's not really the goal
|
||||||
- Shared API contract is
|
- Shared API contract is
|
||||||
- The built-in version will hopefully become great
|
- The built-in version will hopefully become great
|
||||||
- But must be done with careful planning and consideration
|
- But must be done with careful planning and consideration
|
||||||
|
@ -760,25 +759,25 @@ class: flex justify-center text-xl flex-col
|
||||||
<!--
|
<!--
|
||||||
- Being built-in reduces the battier to entry
|
- Being built-in reduces the battier to entry
|
||||||
- Integrating becomes much simpler
|
- Integrating becomes much simpler
|
||||||
- There's 1 API to learn, and it will last you a while
|
- There's 1 API to learn
|
||||||
- Much like the ORM has done for different DB engines
|
- It will last you a while
|
||||||
|
- Scale with your needs
|
||||||
- A developer can join a new project and already be productive
|
- A developer can join a new project and already be productive
|
||||||
- A common API also helps library maintainers
|
- A common API also helps library maintainers
|
||||||
- Maintaining a large library is work enough
|
- Maintaining a large library is work enough
|
||||||
- Without needing to think about how to move code to the background
|
- Without needing to think about how to move code to the background
|
||||||
- If Django can take complexity off you, that's great
|
- If Django can take complexity off you, great
|
||||||
- Currently, it's not really an option
|
- Currently, it's not really an option
|
||||||
- The burden is just too great
|
- The burden is too great
|
||||||
- With this, no additional dependencies
|
- No additional dependencies for your library
|
||||||
- Just import from Django and you're set
|
- Just import from Django and you're set
|
||||||
- The user can use what they want
|
- The user can use what they want
|
||||||
- Or what's suitable for their scale and use case
|
- Or what's suitable for their scale and use case
|
||||||
- Now the barrier is reduced, the ecosystem can flourish
|
- Now the barrier is reduced, the ecosystem can flourish
|
||||||
- Libraries can start assuming background workers, without any additional burden
|
- Libraries can assume background workers, without any additional burden
|
||||||
- The ORM backend should work for the majority of projects
|
- The ORM backend should work for the majority of projects
|
||||||
- If you just want to send emails in the background, you probably don't need Celery
|
- If you just want to send emails in the background, you probably don't need Celery or RQ
|
||||||
- It's overkill
|
- It's overkill
|
||||||
- Even RQ is a bit much
|
|
||||||
- A vendored solution makes it the easiest to get started with
|
- A vendored solution makes it the easiest to get started with
|
||||||
- Tweak some settings, run an extra process, and you're done.
|
- Tweak some settings, run an extra process, and you're done.
|
||||||
-->
|
-->
|
||||||
|
@ -836,12 +835,11 @@ layout: center
|
||||||
- And I've had many times
|
- And I've had many times
|
||||||
- ElasticSearch is quite likely better for the ~10% of people who need it
|
- ElasticSearch is quite likely better for the ~10% of people who need it
|
||||||
- But that doesn't mean the other 90% of people won't be happy with PostgreSQL
|
- But that doesn't mean the other 90% of people won't be happy with PostgreSQL
|
||||||
- And probably wouldn't benefit from ElasticSearch anyway
|
- Probably wouldn't benefit from ElasticSearch anyway
|
||||||
- And definitely won't get a return on the extra hosting cost and complexity
|
- Definitely won't get a return on the extra hosting cost and complexity
|
||||||
- They'll be perfectly happy with Postgres FTS
|
- They'll be perfectly happy with Postgres FTS
|
||||||
- Let them get started the easiest way possible
|
- Let them get started the easiest way possible
|
||||||
- We can still invite them into ElasticSearch when they're ready
|
- We can still invite them into ElasticSearch when they're ready
|
||||||
|
|
||||||
-->
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
|
@ -850,6 +848,10 @@ layout: section
|
||||||
|
|
||||||
# Where are we now?
|
# Where are we now?
|
||||||
|
|
||||||
|
<!--
|
||||||
|
- I mean, other than Vigo
|
||||||
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
layout: image
|
layout: image
|
||||||
image: /dep.png
|
image: /dep.png
|
||||||
|
@ -873,7 +875,7 @@ layout: section
|
||||||
- The ORM backend is where the magic happens
|
- The ORM backend is where the magic happens
|
||||||
- Tell me about all the bugs in my code
|
- Tell me about all the bugs in my code
|
||||||
- The more testing we can do now, the better
|
- The more testing we can do now, the better
|
||||||
- There are still features to implement and improve
|
- There's still work to do
|
||||||
-->
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
|
@ -885,15 +887,14 @@ layout: section
|
||||||
<!--
|
<!--
|
||||||
- More testing
|
- More testing
|
||||||
- Upstreaming
|
- Upstreaming
|
||||||
- The main benefit is in this becoming part of Django
|
- That's the big benefit
|
||||||
- The DEP is approved (ish)
|
- Else it really is just another standard
|
||||||
- Once `django-tasks` is in a better state, it can become `django.tasks`
|
- Once `django-tasks` is in a better state, it can become `django.tasks`
|
||||||
- Hopefully in time for the 5.2 release window
|
- Hopefully in time for the 5.2 release window
|
||||||
- Adoption
|
- Adoption
|
||||||
- The more people know about this, the better it is for everyone
|
- The more people know about this, the better it is for everyone
|
||||||
- Developers can start using `django-tasks` now, and swap for `django.tasks` later
|
- Developers can start working on integrating now
|
||||||
- Or use both side-by-side in older versions of Django / older libraries
|
- Knowing they can trivially upgrade once it's in Django
|
||||||
- The 2 will work correctly together inside the same project
|
|
||||||
-->
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
|
@ -913,12 +914,13 @@ background: /celery.svg
|
||||||
<!--
|
<!--
|
||||||
- Is this the end for Celery and alike?
|
- Is this the end for Celery and alike?
|
||||||
- Not at all!
|
- Not at all!
|
||||||
- If you're using Celery, you've not made a mistake
|
- You've not made a mistake
|
||||||
- It's a great choice
|
- It's a great choice
|
||||||
- They have quite a head start
|
- They have quite a head start
|
||||||
- This is much more about usability and flexibility
|
- This is much more about usability and flexibility
|
||||||
- If you need certain features, keep using them!
|
- If you need certain features, keep using them!
|
||||||
- But, now you have the option of a nice, Django-native API
|
- Now you have the option of a Django-native API
|
||||||
|
- Which could even be Celery under the hood
|
||||||
-->
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
|
@ -941,11 +943,13 @@ class: flex justify-center flex-col text-xl
|
||||||
|
|
||||||
<!--
|
<!--
|
||||||
- The world of background workers is huge
|
- The world of background workers is huge
|
||||||
|
- There are countless nice features
|
||||||
- Not everything is making it into the initial version(s)
|
- Not everything is making it into the initial version(s)
|
||||||
- And that's ok!
|
- And that's ok!
|
||||||
- Existing libraries have a head start
|
- Existing libraries have a head start
|
||||||
- But I hope we can slowly catch them up
|
- But I hope we can slowly catch them up
|
||||||
- Bringing the stability and longevity guarantees that come with Django
|
- Bringing the stability and longevity guarantees that come with Django
|
||||||
|
- Doesn't mean they'll never come
|
||||||
-->
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
|
@ -958,14 +962,16 @@ background: https://images.unsplash.com/photo-1519187903022-c0055ec4036a?q=80&w=
|
||||||
<!--
|
<!--
|
||||||
- The future is bright though
|
- The future is bright though
|
||||||
- In time, I see more and more people reaching to `django.tasks`
|
- In time, I see more and more people reaching to `django.tasks`
|
||||||
|
- And background workers in general
|
||||||
- Moving work to the background will make Django apps _seem_ faster
|
- Moving work to the background will make Django apps _seem_ faster
|
||||||
- Improve throughput
|
- Improve throughput
|
||||||
- Reduce latency
|
- Reduce latency
|
||||||
- Improve reliability
|
- Improve reliability
|
||||||
- Gone are the days of needing additional research and testing to find the tooling you need
|
- Gone are the days of needing additional research and testing to find the tooling you need
|
||||||
- You can use the ones built-in to Django
|
- You can use the ones built-in to Django
|
||||||
- And as you scale, if you find you need to swap something out, you can
|
- And as you scale, it's easy to change
|
||||||
- _without_ rewriting half your application
|
- _without_ rewriting half your application
|
||||||
|
- With all the knowledge to make an informed decision
|
||||||
-->
|
-->
|
||||||
|
|
||||||
---
|
---
|
||||||
|
@ -975,12 +981,7 @@ layout: section
|
||||||
# What's next?
|
# What's next?
|
||||||
|
|
||||||
<div class="absolute right-1/2 translate-x-1/2 mt-12">
|
<div class="absolute right-1/2 translate-x-1/2 mt-12">
|
||||||
<QRCode
|
<img src="/django-tasks-qrcode.png" width="110px" />
|
||||||
:width="120"
|
|
||||||
:height="120"
|
|
||||||
data="https://pypi.org/project/django-tasks/"
|
|
||||||
:dotsOptions="{ color: 'white' }"
|
|
||||||
/>
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!--
|
<!--
|
||||||
|
@ -994,16 +995,36 @@ layout: section
|
||||||
- There's plenty of work to do
|
- There's plenty of work to do
|
||||||
- And I can't do it alone!
|
- And I can't do it alone!
|
||||||
- If you maintain a worker library
|
- If you maintain a worker library
|
||||||
- Or have been burned by one
|
- Or have been burned by one...
|
||||||
- let's chat!
|
|
||||||
-->
|
-->
|
||||||
|
|
||||||
|
---
|
||||||
|
layout: center
|
||||||
|
class: text-center text-xl
|
||||||
|
---
|
||||||
|
|
||||||
|
# Let's chat!
|
||||||
|
|
||||||
|
<ul class="list-none! [&>li]:m-0!">
|
||||||
|
<li><mdi-earth /> theorangeone.net</li>
|
||||||
|
<li><mdi-github /> @RealOrangeOne</li>
|
||||||
|
<li><mdi-twitter /> @RealOrangeOne</li>
|
||||||
|
<li><mdi-mastodon /> @jake@theorangeone.net</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<div class="absolute right-1/2 translate-x-1/2 mt-3">
|
||||||
|
<img src="/dceu24-qrcode.png" width="150px" />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<style>
|
||||||
|
.slidev-layout {
|
||||||
|
background-color: #17181c;
|
||||||
|
color: #e85537;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
|
||||||
---
|
---
|
||||||
layout: end
|
layout: end
|
||||||
---
|
---
|
||||||
|
|
||||||
END
|
END
|
||||||
|
|
||||||
<div class="absolute right-1/2 bottom-4 translate-x-1/2">
|
|
||||||
<img src="/dceu24-qrcode.png" width="110px" />
|
|
||||||
</div>
|
|
||||||
|
|
Loading…
Reference in a new issue