Agenda ● My motivation ● About sidekiq gem ● ainame/Lumpik ● vs Crystal-lang ● Redis client
My motivation ● SSS(Server Side Swift) is very very very hard!! ○ Can’t keep writing without enthusiasm ○ It’s fun for me. ● Learn not only Swift but also the System Programming ○ Swift has compatibility for C-APIs, we can write applications with low-level APIs. ● Be a Hacker ○ Few people try ○ This is an opportunity
SSS except Web ● There are many Web app frameworks in Swift ○ Kitura, Vapor, Zewo, Perfect, etc… ○ https://github.com/swift-server/http ● How about the background job processing? ○ There’s nothing? ○ Opportunity for me
Background job processing ● works with Web app normaly ○ Web app should respond for the client quickly ● works for the time-consuming tasks ○ UPDATE query ○ send push/e-mail notifications ○ video encoding/image processing etc...
Sidekiq ● Fastest Ruby’s job system for general purpose ○ works with Redis as a queue for jobs ○ works with rails well ● Why many rubyists choose sidekiq? ○ less memory & low latency ○ nice web admin ○ Pro/Enterprise Support!! http://sidekiq.org/about
Inside Sidekiq ● Sidekiq has 3 main modules ○ Processor - fetches a job and executes a worker on dedicated thread ○ Poller - poll and enqueue jobs to retry ○ Heatbeat - send statictics metadata to Redis (lib/sidekiq/launcher.rb) ● Definition: Processor/Worker/Job ○ Job is data like arguments of function for Worker ○ Worker is the PORO what you want to do ○ Processor fetches a job and executes
Manager Manage retried jobs ● re-enqueue retry jobs ● adjust polling intervals by the number of processes Processor Worker Job Poller Processor Worker Job Fetch Redis Processor Worker Job Heatbeat Processor Worker Job Send process statuses for the admin ● How many do processors work? ● How many did jobs succeeded/failed? ● What job dose processor work with now?
FIFO queue on the Redis Append a job to light queue > RPUSH “light” 58 id=58 jobs > BRLPOP “light” “heavy” 2 Fetch a job from light id=5 id=7 id=27 …. two queues id=5 > RPUSH “heavy” “ios” “ios” heavy “abc” “swift” “fuga” ….
ainame/Lumpik Sidekiq clone on the Swift ● Fastest Swift’s job queue system for general purpose? ● Type-safe arguments ● Good concurrency with GCD ● Compatibiity for sidekiq-web
Dependencies ● vapor/redis・・・ Redis client ● ainame/Swift-Daemon・・・Damonize a process ● IBM-Swift/BlueSignals・・・Handle signals ● SwiftBeaver/SwiftBeaver・・・Logger ● kylef/Commander・・・Option parser ● jpsim/Yams・・・YAML
ainame/Swift-Daemon ● Daemonize a Swift process ● How to detach a process from the shell 1. fork 2. setsid 3. dup2 ● Learn from ruby/ruby
Invoke a specified worker from String Swift can’t instatiate any class from String → Generate Router class from source code with Sourcery
How to use ainame/Lumpik $ swift package init $ emacs Package.swift $ swift package resolve $ emacs xxxx/main.swift ... $ brew install sourcery $ sourcery --sources xxx --templates Templates --output xxx
ainame/Lumpik is a WIP ● Not implemented yet several features ○ dead queue ○ scheduled queue ● Targets for Swift 4 ○ faster String APIs ○ adopt Codable/JSONEncoder, but Swift doesn’t support it on the Linux currently ● Cloud be faster than sidekiq.cr ○ make faster Redis client in Swift
vs sidekiq.cr Original author, @mperham makes faster sidekiq on the Crystal-lang ● Pros ○ mperham/sidekiq.cr is really fast ○ ruby like syntax ○ very fast ● Cons ○ doesn’t work CPU intensive jobs well by multithreading
What I want for Redis client in Swift ● Prefer method signature to enum based signature for command definition ○ redis.set(“key”, value) vs redis.command(.set, [“key”, value]) ● Prefer real typed value to enum based value for response ○ get(_ key: String) throws -> String vs get(_ key: String) -> ResponseValue ● Pipelining / MULTI-EXEC transaction ○ Pipelining is a basic technique to speedup ● Using buffered IO based TCP client ○ buffered IO is also basic technique to speedup
Recap ● Sidekiq is one of the most successful OSS ● Learn how to make good middleware with 写経 ● Swift dosen’t suit for system programming now ● We have to create fast and easy IO library like Ruby or Crystal