Event Sourcing & Microservice Architectures With Golang
Who am I? • Luke Roberts • Failed Musician • Ruby -> Go • @awsmsrc on the internet • Work for Sqwiggle • Interrupt with questions!
What is Sqwiggle? • WebRTC based video collaboration platform for the web and mac • Monolithic architecture built on RoR • 3 apps backed by the same datastore • Modeled via ActiveRecord backed by PostgreSQL
So why am I here? • • Side Project • More passive • Full Disclosure - Small Scale Private Beta • Ask for an invite! • Reusable components for the next experiment required
Great so we can reuse all the codez??
Problems? • New developers need to conceptualize the whole system • Running the full test suite took upwards of 10 minutes • Deployments were closer to 20 minutes • Development boot time was approaching a minute • Even small changes required a deployment of the entire application • Each dependency (redis, memcache) was a dependency for everyone/everything • The haystack was large so tracking down bugs was often difficult • Conflicts between front and backend developers were common • Performance as perceived by the end user • Real time app, doesn't fit classic web development paradigms
Our solution • Microservices • EventSourcing • CQRS • Asynchronous Message Broker • I’ll buy you a pint after the talk in exchange for more buzzwords I can take to my boss!
What are Microservices • Discrete applications that collaborate towards a common goal • Communication is often via HTTP(S)/REST • Can be deployed on the same or different machines • Have their own data stores • Can be scaled vertically or horizontally dependent on functionality
What is Event Sourcing • Store every event that mutates current application state • Workers/Modules/Services subscribe to events and cache what they need • Store snapshots of current state • Always have *Eventual Consistency* in mind
Advantages of Event Sourcing • Complete log of every state change ever • Trace/Debug every interaction within the system • Replicate the system at any point in time • Great performance thanks to parallelization • No more ORM’s YAY!!! • Data Mining
CQRS as a side effect • Command Query Responsibility Segregation • Often lumped in with event sourcing • Avoid giant model types (less of a problem in Go) • E.G User creation validation rules run in a separate process to the CRM interface shown to the non tech people
Issues with Event Sourcing • Side-effects when replaying events - ( Welcome emails ;) ) • Querying the event store for reports and overviews • Changing event formats or payloads • Concurrent writes
Events lifecycle • This is our own naive approach YMMV • Present tense is a request or command • Past tense is an event that signals new application state
Request - (optionally saved to event store)
Response - (Saved to event store)
Error Response - (optionally saved to event store)
RabbitMQ - The glue that holds it all together • Persistent queue based message broker • Acts as a load balancer • Well supported by all viable languages and platforms • Flexible topologies, ideal for fanning events out through services • Alternatives: Redis / PgSQL pub-sub/ sidekiq / beanstalkd / IronMQ / Amazon SQS • Currently have 3 exchanges: Events, Timber & Services
So why not HTTP/REST • HTTP(S) is slow • Transport vs application responsibilities are often poorly defined • Synchronous (Not ideal for soft real time multi media apps)
EXAMPLE: Invite creation • Client: emits invite.create on the events exchange • Karaoke: validates and authorizes • Karaoke: emits invite.created on the events exchange (does not store the invite) • Timber: stores the event and emits invite.created on the timber exchange • Karaoke: receives the invite.created on the timber exchange and updates its private/internal state and sends an email • Auth Service: receives the invite.created on the timber exchange and creates a record of the invite code to allow for account creation
Why is Go a great fit? • Simple • Fast • Easy to deploy, runs anywhere • Asynchronous message processing • Micro Services are often I/O heavy/bound
Caveats & tradeoffs • Complex • Data aggregation and reporting • More data storage is needed (excluding files and other assets) • Often a different approach to previous experience • Lack of community/blog posts/libraries
Development with Docker & Compose • Developers can exactly replicate the environment • Developers can start and stop only the services they need: $: docker-compose start karaoke • Environment is under version control • No need for virtual machines
Development with Docker & Compose
Deployment & Orchestration • Currently Ansible • Rolling deployments with Docker not solved • Flynn, Dokku, Deis and many others • Docker are rolling their own orchestration • If your using docker in production, lets chat!
In Summary • If its a CRUD app, use Rails, Django & break off Go services as necessary • Complexity CAN be a one to one mapping for flexibility • State has bugs, a history of events is concrete even if assumptions on the world are wrong/change • Go is GREAT for micro services
You should consider Event Sourcing if…. • Your app doesn't naturally map with the CRUD paradigm • You need version history for audits or ‘undo’ actions • Your application is highly concurrent with long lived operations/actions • You're scaling and growth patterns are unclear