Chained Tasks •Batch of 10,000 Followers Per Task •Tasks Yield Successive Tasks •Much Finer-Grained Load Balancing •Failure/Reload Penalty Low
Other Async Tasks •Cross-Posting to Other Networks •Search Indexing •Spam Analysis •Account Deletion •API Hook
In the beginning...
Gearman & Python •Simple, Purpose-Built Task Queue •Weak Framework Support •We just built ad hoc worker scripts •A mess to add new job types & capacity
Gearman in Production •Persistence horrifically slow, complex •So we ran out of memory and crashed, no recovery •Single core, didn’t scale well: 60ms mean submission time for us •Probably should have just used Redis
We needed a fresh start.
WARNING System had to be in production before the heat death of the universe. We are probably doing something stupid!
Redis •We Already Use It •Very Fast, Efficient •Polling For Task Distribution •Messy Non-Synchronous Replication •Memory Limits Task Capacity
Beanstalk • Purpose-Built Task Queue • Very Fast, Efficient • Pushes to Consumers • Spills to Disk • No Replication • Useless For Anything Else
RabbitMQ • Reasonably Fast, Efficient • Spill-To-Disk • Low-Maintenance Synchronous Replication • Excellent Celery Compatibility • Supports Other Use Cases • We don’t know Erlang
Our RabbitMQ Setup •RabbitMQ 3.0 •Clusters of Two Broker Nodes, Mirrored •Scale Out By Adding Broker Clusters •EC2 c1.xlarge, RAID instance storage •Way Overprovisioned
Alerting •We use Sensu •Monitors & alerts on queue length threshold •Uses rabbitmqctl list_queues
Graphing •We use graphite & statsd •Per-task sent/fail/success/retry graphs •Using celery's hooks to make them possible
workers web 0A 1A 2A us-east-1a us-east-1e 0E 1E 2E
Mean vs P90 Publish Times (ms)
Tasks per second
Aggregate CPU% (all RabbitMQs)
Wait, ~4000 tasks/sec... I thought you said scale?
~25,000 app threads publishing tasks
Celery IRL •Easy to understand, new engineers come up to speed in 15 minutes. •New job types deployed without fuss. •We hack the config a bit to get what we want.
Related tasks run on the same queue @task(routing_key="task_queue") def task_function(task_arg, another_task_arg): do_things()
Scaling Out • Celery only supported 1 broker host last year when we started. • Created kombu-multibroker "shim" • Multiple brokers used in a round-robin fashion. • Breaks some Celery management tools :(
gevent = Network Bound •Facebook API •Tumblr API •Various Background S3 Tasks •Checking URLs for Spam
Problem: Network-Bound Tasks Sometimes Need To Take Some Action
Ran on "gevent" worker @task(routing_key="task_remote_access"): def check_url(object_id, url): is_bad = run_url_check(url) if is_bad: take_some_action.delay(object_id, url) @task(routing_key="task_action"): def take_some_action(object_id, url): do_some_database_thing() Ran on "processes" worker
Problem: Slow Tasks Monopolize Workers
Fetches Batch Main 5 Worker 4 5 4 3 2 1 0 3 Broker 2 0 Worker 1 0 Worker 1 Wait Until Batch Finishes Before Grabbing Another One
•Run higher concurrency? Inefficient :( •Lower batch (prefetch) size? Min is concurrency count, inefficient :( •Separate slow & fast tasks :)
Why not do this everywhere? •Tasks must be idempotent! •That probably is the case anyway :( •Mirroring can cause duplicate tasks •FLP Impossibility FFFFFFFFFUUUUUUUUU!!!!
There is no such thing as running tasks exactly-once.
"... it is impossible for one process to tell whether another has died (stopped entirely) or is just running very slowly." Impossibility of Distributed Consensus with One Faulty Process Fischer, Lynch, Patterson (1985)
NLP Proof Gives Us Choices: To retry or not to retry
Problem: Early on, we noticed overloaded brokers were dropping tasks...
Publisher Confirms •AMQP default is that we don't know if things were published or not. :( •Publisher Confirms makes broker send acknowledgements back on publishes. •kombu-multibroker forces this. •Can cause duplicate tasks. (FLP again!)
Other Rules of Thumb
Avoid using async tasks as a "backup" mechanism only during failures. It'll probably break.
Only pass self-contained, non-opaque data (strings, numbers, arrays, lists, and dicts) as arguments to tasks. @task(routing_key="media_activation") def deactivate_media_content(media_id): try: media = get_media_store_object(media_id) media.deactivate() except MediaContentRemoteOperationError, e: raise deactivate_media_content.retry(countdown=60)
Tasks should usually execute within a few seconds. They gum up the works otherwise.